Image segmentation method and model training method and related device, electronic equipment

Abstract

The present application discloses an image segmentation method and a training method of its model and related devices and electronic devices, wherein the training method of the image segmentation model includes: obtaining medical image samples including a first medical image set annotated with at least one target object and a second medical image set without annotations; using the first image segmentation model to perform segmentation processing on the first medical image set, obtaining a first segmentation result corresponding to the first medical image set, and determining a first loss function of the first image segmentation model; using the first image segmentation model and a second image segmentation model obtained by sliding average of its network parameters to perform segmentation processing on the second medical image set respectively, obtaining corresponding second segmentation results and third segmentation results, and determining a second loss function of the first image segmentation model; adjusting the network parameters of the image segmentation model according to the first loss function and the second loss function. The above scheme can obtain a highly robust target object segmentation result.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to an image segmentation method and a model training method thereof, and related devices and electronic equipment. Background Art

[0002] The segmentation of organs, blood vessels and other target objects in medical images such as CT (Computed Tomography) and MRI (Magnetic Resonance Imaging) is of great clinical significance. For example, high-precision and high-robustness abdominal multi-organ segmentation is conducive to computer-aided diagnosis and computer-aided surgical planning.

[0003] However, in actual research, there are several common problems: there is an inconsistency in the labeling of multi-center data sets; there are a large number of organs with different sizes and shapes; accurate multi-organ labeling consumes huge manpower and material resources, but unlabeled data is massive and easy to obtain. In view of this, how to improve the versatility of image segmentation has become a topic of great research value. Summary of the invention

[0004] The present application provides an image segmentation method and a model training method thereof, as well as related devices and electronic equipment.

[0005] In a first aspect, the present application provides a training method for an image segmentation model, which is applied to a first image segmentation model and a second image segmentation model, wherein the second image segmentation model is obtained by sliding average of network parameters of the first image segmentation model; the training method comprises: obtaining medical image samples; wherein the medical image samples comprise a first medical image set annotated with at least one target object and a second medical image set without annotations; performing segmentation processing on the first medical image set using the first image segmentation model to obtain a first segmentation result corresponding to the first medical image set; determining a first loss function of the first image segmentation model based on the first segmentation result; performing segmentation processing on the second medical image set using the first image segmentation model and the second image segmentation model respectively to obtain a second segmentation result corresponding to the second medical image set segmented by the first image segmentation model, and a third segmentation result corresponding to the second medical image set segmented by the second image segmentation model; determining a second loss function of the first image segmentation model based on the second segmentation result and the third segmentation result; adjusting the network parameters of the first image segmentation model using the first loss function and the second loss function, and adjusting the network parameters of the second image segmentation model based on the adjusted network parameters of the first image segmentation model.

[0006] Therefore, by obtaining medical image samples including a first medical image set labeled with at least one target object and a second unlabeled medical image set, and using the first image segmentation model to perform segmentation processing on the first medical image set, a first segmentation result corresponding to the first medical image set can be obtained, and then a first loss function of the first image segmentation model can be determined based on the first segmentation result. In addition, the second medical image set is segmented using the first image segmentation model and the second image segmentation model respectively, and then a second segmentation result corresponding to the second medical image set segmented by the first image segmentation model and a third segmentation result corresponding to the second medical image set segmented by the second image segmentation model can be obtained. Since the second image segmentation model is obtained by sliding average of the network parameters of the first image segmentation model, the second loss function of the first image segmentation model can be determined based on the second segmentation result and the third segmentation result. Therefore, the labeled first medical image set and the unlabeled second medical image set are combined together, and then trained together in a semi-supervised manner, a highly robust target object segmentation result can be obtained.

[0007] Wherein, the obtaining of medical image samples includes: performing balancing processing on each collected medical image set so that the proportion of medical images of each period in the first medical image set is the same, and / or the proportion of medical images of each period in the second medical image set is the same.

[0008] Therefore, considering that the medical images that make up each medical image set come from different data sets, and the annotations of each data set are different, and the scanning machine parameters and the quality of the medical images used are also different, by balancing each collected medical image set, the proportion of medical images of each period in the first medical image set is the same, and / or the proportion of medical images of each period in the second medical image set is the same, and training is performed using the balanced medical image samples, it can be ensured that the trained image segmentation model can be applicable to multi-center medical image data sets with non-uniform annotations, and has good versatility.

[0009] Before determining the first loss function of the first image segmentation model based on the first segmentation result, the method includes: generating a loss function weight for each target object based on the annotation information of the first medical image set and the volume prior information of each target object; wherein the volume prior information of each target object includes the volume mean of all annotations corresponding to the target object; determining the first loss function of the first image segmentation model based on the first segmentation result includes: calculating the first loss function based on the first segmentation result and the loss function weight of each target object.

[0010] Therefore, according to the volume prior information of each target object and the annotation information of the first medical image set, the weight of each target object in the loss function can be balanced and controlled, so that it is possible to simultaneously train multi-center, partially annotated medical image sets, making it easy for the image segmentation model to converge.

[0011] Among them, the first loss function includes a three-dimensional segmentation loss function; the first loss function is obtained by calculating the loss function weight based on the first segmentation result and each target object, including: analyzing the difference information between the first segmentation result and the annotation information in the first medical image set, and calculating the three-dimensional segmentation loss function of the first image segmentation model according to the loss function weight of each target object.

[0012] Therefore, by analyzing the difference information between the first segmentation result and the annotation information in the first medical image set, and according to the loss function weight of each target object, the three-dimensional segmentation loss function of the first image segmentation model can be calculated. Then, the segmentation loss function can be calculated in three-dimensional space, so that the image segmentation model has higher segmentation performance.

[0013] Among them, the first loss function also includes a two-dimensional projection boundary loss function; the first loss function is obtained by calculating the loss function weight based on the first segmentation result and each target object, and also includes: projecting the first segmentation result onto a two-dimensional plane, and obtaining the segmentation boundary information of each target object according to the projection result; calculating the boundary loss of each target object on the two-dimensional projection through the segmentation boundary information, and obtaining the two-dimensional projection boundary loss function of the first image segmentation model.

[0014] Therefore, by projecting the first segmentation result onto a two-dimensional plane, the segmentation boundary information of each target object is obtained according to the projection result, and then the boundary loss of each target object on the two-dimensional projection is calculated through the segmentation boundary information, and the two-dimensional projection boundary loss function of the first image segmentation model is obtained. Therefore, in addition to calculating the segmentation loss function in three-dimensional space, the boundary loss of each target object on the two-dimensional projection is also calculated synchronously on the two-dimensional level. Without introducing too much calculation amount, the segmentation result can be better constrained in spatial position, thereby improving the accuracy of boundary segmentation of each target object, greatly improving the overall robustness of image segmentation, and can accelerate the optimization of the image segmentation model, and improve the segmentation performance and generalization ability of the image segmentation model.

[0015] Among them, the use of the first image segmentation model and the second image segmentation model to respectively perform segmentation processing on the second medical image set to obtain a second segmentation result corresponding to the second medical image set segmented by the first image segmentation model, and a third segmentation result corresponding to the second medical image set segmented by the second image segmentation model, includes: adding a first random noise to the second medical image set and inputting it into the first image segmentation model to obtain the second segmentation result, adding a second random noise to the second medical image set and inputting it into the second image segmentation model to obtain the third segmentation result; the second loss function includes a segmentation consistency loss function; and determining the second loss function of the first image segmentation model based on the second segmentation result and the third segmentation result includes: analyzing the difference information between the second segmentation result and the third segmentation result to obtain the segmentation consistency loss function of the first image segmentation model.

[0016] Therefore, a semi-supervised training strategy is adopted, and the unlabeled second medical image set is respectively input into the first image segmentation model as the "student" model and the second image segmentation model as the "teacher" model to obtain the corresponding second segmentation results and the third segmentation results. Then, by analyzing the difference information between the second segmentation results and the third segmentation results, the segmentation consistency loss function of the first image segmentation model and the second image segmentation model can be obtained. Therefore, the image segmentation model can be optimized through the segmentation consistency loss function, so that the image segmentation model can generate more stable segmentation results, that is, a large amount of unlabeled medical image data is used to improve the robustness of the image segmentation model.

[0017] Among them, the first loss function and the second loss function are used to adjust the network parameters of the first image segmentation model, and the network parameters of the second image segmentation model are adjusted based on the adjusted network parameters of the first image segmentation model; including: transmitting the first loss function and the second loss function to the first image segmentation model through a reverse transmission algorithm to adjust the network parameters of the first image segmentation model; and performing sliding averaging on the adjusted network parameters of the first image segmentation model to obtain the adjusted network parameters of the second image segmentation model.

[0018] Therefore, the first loss function and the second loss function are transmitted to the first image segmentation model through the back-transfer algorithm, and the network parameters of the first image segmentation model can be adjusted to optimize the first image segmentation model. Then, the network parameters of the adjusted first image segmentation model are sliding averaged to obtain the adjusted network parameters of the second image segmentation model, that is, the second image segmentation model is optimized. Therefore, the first medical image set labeled with at least one target object and a large number of unlabeled second medical image sets are fully utilized, so that the first image segmentation model as a "student" model and the second image segmentation model as a "teacher" model can be jointly optimized and improved to improve the robustness and segmentation performance of the image segmentation model.

[0019] In order to solve the above-mentioned problem, the second aspect of the present application provides an image segmentation method, which includes: obtaining a medical image to be segmented; using a first image segmentation model and / or a second image segmentation model to perform segmentation processing on the medical image to be segmented, and obtaining a segmentation result corresponding to the medical image to be segmented; wherein the first image segmentation model and the second image segmentation model are obtained using the training method of the image segmentation model in the above-mentioned first aspect.

[0020] In order to solve the above problems, the third aspect of the present application provides a training device for an image segmentation model, the training device comprising: a sample acquisition module, used to acquire medical image samples; wherein the medical image samples include a first medical image set annotated with at least one target object and a second medical image set without annotations; an image segmentation module, used to segment the first medical image set using the first image segmentation model to obtain a first segmentation result corresponding to the first medical image set; and, using the first image segmentation model and the second image segmentation model to segment the second medical image set respectively, to obtain a second segmentation result corresponding to the second medical image set segmented by the first image segmentation model, and a third segmentation result corresponding to the second medical image set segmented by the second image segmentation model; wherein the second image segmentation model is obtained by sliding average of the network parameters of the first image segmentation model; a loss function determination module, used to determine the first loss function of the first image segmentation model based on the first segmentation result; and, based on the second segmentation result and the third segmentation result, determine the second loss function of the first image segmentation model; a parameter adjustment module, used to adjust the network parameters of the first image segmentation model using the first loss function and the second loss function, and adjust the network parameters of the second image segmentation model based on the adjusted network parameters of the first image segmentation model.

[0021] In order to solve the above problems, the fourth aspect of the present application provides an image segmentation device, which includes: an image acquisition module for acquiring a medical image to be segmented; an image segmentation module for using a first image segmentation model and / or a second image segmentation model to perform segmentation processing on the medical image to be segmented to obtain a segmentation result corresponding to the medical image to be segmented; wherein the first image segmentation model and the second image segmentation model are obtained using the training method of the image segmentation model in the above-mentioned first aspect.

[0022] In order to solve the above problems, the fifth aspect of the present application provides an electronic device, comprising a memory and a processor coupled to each other, wherein the processor is used to execute program instructions stored in the memory to implement the image segmentation model training method in the above first aspect, or the image segmentation method in the above second aspect.

[0023] In order to solve the above-mentioned problem, the sixth aspect of the present application provides a computer-readable storage medium on which program instructions are stored. When the program instructions are executed by a processor, the training method of the image segmentation model in the above-mentioned first aspect, or the image segmentation method in the above-mentioned second aspect is implemented.

[0024] The above scheme obtains medical image samples including a first medical image set labeled with at least one target object and a second unlabeled medical image set, and uses the first image segmentation model to perform segmentation processing on the first medical image set, so as to obtain a first segmentation result corresponding to the first medical image set, and then determine the first loss function of the first image segmentation model based on the first segmentation result. In addition, the first image segmentation model and the second image segmentation model are used to perform segmentation processing on the second medical image set respectively, so as to obtain a second segmentation result corresponding to the second medical image set segmented by the first image segmentation model, and a third segmentation result corresponding to the second medical image set segmented by the second image segmentation model. Since the second image segmentation model is obtained by sliding average of the network parameters of the first image segmentation model, the second loss function of the first image segmentation model can be determined based on the second segmentation result and the third segmentation result. Therefore, the labeled first medical image set and the unlabeled second medical image set are combined together, and then trained together in a semi-supervised manner, so as to obtain a highly robust target object segmentation result. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 It is a flowchart of an embodiment of a training method for an image segmentation model of the present application;

[0026] Figure 2 It is a flowchart of another embodiment of the training method of the image segmentation model of the present application;

[0027] Figure 3a yes Figure 2 A schematic diagram of a flow chart of an embodiment of step S24;

[0028] Figure 3b It is a schematic diagram of a segmentation model for an application scenario of a training method for an image segmentation model of the present application;

[0029] Figure 4 It is a flowchart of an embodiment of the image segmentation method of the present application;

[0030] Figure 5 It is a schematic diagram of the framework of an embodiment of a training device for an image segmentation model of the present application;

[0031] Figure 6 It is a schematic diagram of the framework of an embodiment of the image segmentation device of the present application;

[0032] Figure 7 It is a schematic diagram of the framework of an embodiment of the electronic device of the present application;

[0033] Figure 8 It is a schematic diagram of a framework of an embodiment of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0034] The scheme of the embodiment of the present application is described in detail below in conjunction with the drawings of the specification.

[0035] In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures, interfaces, and technologies are provided to facilitate a thorough understanding of the present application.

[0036] The terms "system" and "network" are often used interchangeably in this article. The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship. In addition, "many" in this article means two or more than two.

[0037] See also Figure 1 , Figure 1It is a flow chart of an embodiment of the training method of the image segmentation model of the present application. Specifically, the training method of the image segmentation model is applied to a first image segmentation model and a second image segmentation model, the second image segmentation model is obtained by sliding average of the network parameters of the first image segmentation model, the first image segmentation model and the second image segmentation model are used to segment the target objects in the medical image, the medical image may be a three-dimensional image of the abdomen, the corresponding target objects include but are not limited to the liver, left kidney, right kidney, spleen, inferior vena cava, aorta, stomach, gallbladder, etc., and the first image segmentation model and the second image segmentation model can be used to segment and mark multiple organs in the three-dimensional image of the abdomen. The training method of the image segmentation model may include the following steps:

[0038] Step S11: Obtain medical image samples, wherein the medical image samples include a first medical image set annotated with at least one target object and a second medical image set without annotations.

[0039] Specifically, the first image segmentation model and the second image segmentation model are pre-constructed, and medical image samples are used to train the first image segmentation model and the second image segmentation model; taking the abdominal three-dimensional image as an example, the first medical image set contains both the first abdominal image and the second abdominal image, while the second medical image set only contains the second abdominal image, wherein the first abdominal image carries annotations, while the second abdominal image carries no annotations.

[0040] In one implementation scenario, the above step S11 includes: performing a balancing process on each collected medical image set so that the ratio of medical images of each phase in the first medical image set is the same, and / or the ratio of medical images of each phase in the second medical image set is the same. It can be understood that when acquiring medical images, the upper and lower boundaries of multiple organs can be located by designing a boundary regression model, for example, the region of interest of multiple abdominal organs can be intercepted, and invalid areas can be eliminated, thereby avoiding interference of medical images on the image segmentation model. Considering that the medical images that make up each medical image set come from different data sets, and the annotations of each data set are different, and the scanning machine parameters and the quality of the medical images used are also different, each collected medical image set can be balanced to make the proportion of medical images of each period in the first medical image set the same, and the proportion of medical images of each period in the second medical image set the same. For example, the first medical image set and the second medical image set both contain image data of the plain scan period, arterial period, portal period, and delayed period in equal proportions. Therefore, by using the balanced medical image samples for training, it can be ensured that the trained image segmentation model can be applicable to multi-center medical image data sets with inconsistent annotations, and has good versatility.

[0041] Step S12: using the first image segmentation model to perform segmentation processing on the first medical image set to obtain a first segmentation result corresponding to the first medical image set.

[0042] Step S13: Based on the first segmentation result, determine a first loss function of the first image segmentation model.

[0043] It can be understood that by using the first image segmentation model to perform segmentation processing on the first medical image set, a first segmentation result corresponding to the first medical image set can be obtained. Since the first medical image set contains a first abdominal image with annotations, the first loss function of the first image segmentation model can be determined by comparing the first segmentation result corresponding to the first medical image set and the first abdominal image with annotations in the first medical image set.

[0044] Step S14: Use the first image segmentation model and the second image segmentation model to perform segmentation processing on the second medical image set respectively, to obtain a second segmentation result corresponding to the second medical image set segmented by the first image segmentation model, and a third segmentation result corresponding to the second medical image set segmented by the second image segmentation model.

[0045] Step S15: Determine a second loss function of the first image segmentation model based on the second segmentation result and the third segmentation result.

[0046] It can be understood that by using the first image segmentation model and the second image segmentation model to perform segmentation processing on the second medical image set respectively, a second segmentation result corresponding to the second medical image set segmented by the first image segmentation model and a third segmentation result corresponding to the second medical image set segmented by the second image segmentation model can be obtained; since the second image segmentation model is obtained by sliding average of the network parameters of the first image segmentation model, it is expected that the second segmentation results and the third segmentation results predicted by the two have a high consistency, so by comparing the second segmentation results and the third segmentation results, the second loss function of the first image segmentation model can be determined.

[0047] Step S16: using the first loss function and the second loss function, adjusting the network parameters of the first image segmentation model, and adjusting the network parameters of the second image segmentation model based on the adjusted network parameters of the first image segmentation model.

[0048] It can be understood that after obtaining the first loss function and the second loss function, the network parameters of the first image segmentation model can be adjusted according to the first loss function and the second loss function to update the first image segmentation model, and then the updated second image segmentation model can be obtained by slidingly averaging the network parameters of the updated first image segmentation model.

[0049] In addition, during the training process of the image segmentation model, the convergence of the first loss function and the second loss function can be obtained. When the first loss function and the second loss function converge, the updating of the network parameters of the image segmentation model can be stopped. When the first loss function and the second loss function do not converge, the number of adjustments of the network parameters can be obtained. When the number of adjustments reaches a preset number, the final image segmentation model can be determined based on the network parameters at this time to avoid the loss function not converging and affecting the training efficiency.

[0050] In one embodiment, the above step S16 specifically includes: transmitting the first loss function and the second loss function to the first image segmentation model through a reverse transmission algorithm to adjust the network parameters of the first image segmentation model; performing sliding average on the adjusted network parameters of the first image segmentation model to obtain the adjusted network parameters of the second image segmentation model. It can be understood that by transmitting the first loss function and the second loss function to the first image segmentation model through a reverse transmission algorithm, the network parameters of the first image segmentation model can be adjusted to optimize the first image segmentation model, and then the network parameters of the adjusted first image segmentation model are performed on a sliding average to obtain the adjusted network parameters of the second image segmentation model, that is, the second image segmentation model is optimized, so that the first medical image set annotated with at least one target object and a large number of unannotated second medical image sets are fully utilized, so that the first image segmentation model as a "student" model and the second image segmentation model as a "teacher" model can be jointly optimized and improved to improve the robustness and segmentation performance of the image segmentation model.

[0051] The above scheme obtains medical image samples including a first medical image set labeled with at least one target object and a second unlabeled medical image set, and uses the first image segmentation model to perform segmentation processing on the first medical image set, so as to obtain a first segmentation result corresponding to the first medical image set, and then determine the first loss function of the first image segmentation model based on the first segmentation result. In addition, the first image segmentation model and the second image segmentation model are used to perform segmentation processing on the second medical image set respectively, so as to obtain a second segmentation result corresponding to the second medical image set segmented by the first image segmentation model, and a third segmentation result corresponding to the second medical image set segmented by the second image segmentation model. Since the second image segmentation model is obtained by sliding average of the network parameters of the first image segmentation model, the second loss function of the first image segmentation model can be determined based on the second segmentation result and the third segmentation result. Therefore, the labeled first medical image set and the unlabeled second medical image set are combined together, and then trained together in a semi-supervised manner, so as to obtain a highly robust target object segmentation result.

[0052] See also Figure 2 , Figure 2 1 is a flow chart of another embodiment of the training method of the image segmentation model of the present application. Specifically, the training method of the image segmentation model may include the following steps:

[0053] Step S21: Obtain medical image samples, wherein the medical image samples include a first medical image set annotated with at least one target object and a second medical image set without annotations.

[0054] Step S22: Generate a loss function weight for each target object based on the annotation information of the first medical image set and the volume prior information of each target object; wherein the volume prior information of each target object includes the volume mean of all annotations corresponding to the target object.

[0055] Specifically, the first medical image set includes both a first abdominal image with annotations and a second abdominal image without annotations. The annotation information of the first medical image set includes label values ​​Label for each target object. i , where the label value of the labeled target object is 1, the label value of the unlabeled target object is 0, i represents the index of the target object to be segmented, and the average value of all labeled volume data of a target object in the first medical image set is taken as the volume prior information V of the target object. i Therefore, according to the label value Label of each target object i and volume prior information V i , can generate the loss function weight W unique to each target object in different phase data i , the loss function weight of each target object can be used to calculate the loss function, where W i The calculation formula is as follows:

[0056]

[0057] According to formula (1), it can be found that the loss function weight of the target object with a larger volume is relatively low, while the loss function weight of the target object with a smaller volume is relatively high.

[0058] Step S23: using a first image segmentation model to perform segmentation processing on the first medical image set to obtain a first segmentation result corresponding to the first medical image set.

[0059] In this embodiment, step S21 and step S23 are substantially similar to step S11 and step S12 in the above-mentioned embodiment of the present application, and are not described in detail here.

[0060] Step S24: Calculate the first loss function based on the first segmentation result and the loss function weight of each target object.

[0061] It can be understood that based on the volume prior information of each target object and the annotation information of the first medical image set, the weight of each target object in the loss function can be balanced and controlled, so that it is possible to simultaneously train multi-center, partially annotated medical image sets, making it easy for the image segmentation model to converge.

[0062] See also Figure 3a , Figure 3a yes Figure 2 Schematic diagram of a flow chart of an embodiment of step S24 in the embodiment. In this embodiment, the first loss function includes a three-dimensional segmentation loss function and a two-dimensional projection boundary loss function, and the second loss function includes a segmentation consistency loss function; the above step S24 may specifically include the following steps:

[0063] Step S241: analyzing the difference information between the first segmentation result and the annotation information in the first medical image set, and calculating the three-dimensional segmentation loss function of the first image segmentation model according to the loss function weight of each target object.

[0064] Specifically, the three-dimensional segmentation loss function may include the dice coefficient loss function l dice_3d And the cross entropy loss function l cross_entropy_3d etc. are mainstream segmentation loss functions for three-dimensional image segmentation. It can be understood that by analyzing the difference information between the first segmentation result and the annotation information in the first medical image set, and according to the loss function weight of each target object, the three-dimensional segmentation loss function of the first image segmentation model can be calculated, so that the segmentation loss function can be calculated in the three-dimensional space, so that the image segmentation model has a higher segmentation performance.

[0065] Step S242: Project the first segmentation result onto a two-dimensional plane, and obtain segmentation boundary information of each target object according to the projection result.

[0066] Step S243: Calculate the boundary loss of each target object on the two-dimensional projection through the segmentation boundary information to obtain the two-dimensional projection boundary loss function of the first image segmentation model.

[0067] Considering the above dice coefficient loss function l dice_3d And the cross entropy loss function l cross_entropy_3dThere is no constraint on the segmentation boundary itself, and the Hausdorff distance, as a measure of shape similarity, can make a good supplement. However, calculating the Hausdorff distance in the three-dimensional segmentation space will take up a lot of computing resources. Therefore, the idea of ​​high-dimensional projection to low-dimensional can be adopted. The three-dimensional segmentation result (that is, the first segmentation result) is projected along three planes (ZY plane, ZX plane and YX plane) to obtain three two-dimensional projection results. Then, the segmentation boundary information of each target object is obtained on the three two-dimensional projection results. Therefore, the boundary loss of each target object on the two-dimensional projection can be calculated respectively, and the two-dimensional projection boundary loss function l of the first image segmentation model is obtained. project_2d , Hausdorff distance is used to introduce the two-dimensional projection boundary loss function l project_2d , so that the image segmentation model can better learn the spatial distribution characteristics of each target object, thereby improving the accuracy of the segmentation boundary of the target object and greatly reducing the situation where there are "islands" in the segmentation.

[0068] It can be understood that by projecting the first segmentation result onto a two-dimensional plane, the segmentation boundary information of each target object is obtained according to the projection result, and then the boundary loss of each target object on the two-dimensional projection is calculated by the segmentation boundary information, and the two-dimensional projection boundary loss function of the first image segmentation model is obtained. Therefore, in addition to calculating the segmentation loss function in three-dimensional space, the boundary loss of each target object on the two-dimensional projection is also calculated synchronously on the two-dimensional level. Without introducing too much calculation amount, the segmentation result can be better constrained in spatial position, thereby improving the accuracy of boundary segmentation of each target object, greatly improving the overall robustness of image segmentation, and can accelerate the optimization of the image segmentation model, and improve the segmentation performance and generalization ability of the image segmentation model.

[0069] Step S25: adding first random noise to the second medical image set and inputting the result into the first image segmentation model to obtain the second segmentation result; adding second random noise to the second medical image set and inputting the result into the second image segmentation model to obtain the third segmentation result.

[0070] Step S26: Analyze the difference information between the second segmentation result and the third segmentation result to obtain the segmentation consistency loss function of the first image segmentation model.

[0071] Therefore, a semi-supervised training strategy is adopted. After adding random noise to the unlabeled second medical image set, the images are input into the first image segmentation model as the "student" model and the second image segmentation model as the "teacher" model respectively to obtain the corresponding second segmentation results and the third segmentation results. Then, by analyzing the difference information between the second segmentation results and the third segmentation results, the segmentation consistency loss function l of the first image segmentation model and the second image segmentation model can be obtained. consistency, so by segmenting the consistency loss function l consistency The image segmentation model can be optimized so that it can generate more stable segmentation results, that is, a large amount of unlabeled medical image data can be used to improve the robustness of the image segmentation model.

[0072] Step S27: using the first loss function and the second loss function, adjusting the network parameters of the first image segmentation model, and adjusting the network parameters of the second image segmentation model based on the adjusted network parameters of the first image segmentation model.

[0073] Therefore, the first loss function finally obtained includes the three-dimensional segmentation loss function (the dice coefficient loss function l dice_3d And the cross entropy loss function l cross_entropy_3d ) and the two-dimensional projection boundary loss function l project_2d , the second loss function includes the segmentation consistency loss function l consistency , so the network parameters of the first image segmentation model as the "student" model can be continuously updated and optimized by minimizing the loss function using the gradient descent method, where the loss function Loss of the first image segmentation model satisfies formula (2):

[0074] Loss = l dice_3d +l cross_entropy_3d +l consistency +l project_2d (2)

[0075] In this embodiment, step S27 is basically similar to step S16 in the above embodiment of the present application and will not be repeated here.

[0076] Please combine Figure 3b , Figure 3b This is a schematic diagram of a segmentation model for an application scenario of the training method of the image segmentation model of the present application. In the training process of the image segmentation model, the volume prior information of each target object and the annotation information of a part of the annotated data set are used to train the student model, so as to balance and control the weight of each target object in the loss function, and calculate the segmentation loss function (the dice coefficient loss function l dice_3d And the cross entropy loss function l cross_entropy_3d ), so that the image segmentation model has a higher segmentation performance, and the boundary loss of each target object on the two-dimensional projection is calculated synchronously on the two-dimensional level (the two-dimensional projection boundary loss function l project_2d), can better constrain the spatial position of the segmentation results, thereby improving the accuracy of the boundary segmentation of each target object and improving the segmentation performance and generalization ability of the image segmentation model. In addition, in the training process of the image segmentation model, in addition to using some labeled data sets for training, a large number of unlabeled multi-organ data sets are also used. After adding random noise to the unlabeled data sets, they are input into the "student" model and the "teacher" model respectively to obtain the corresponding segmentation results. Then, by analyzing the difference information between the segmentation results corresponding to the "student" model and the segmentation results corresponding to the "teacher" model, the segmentation consistency loss function l of the image segmentation model can be obtained. consistency , so by segmenting the consistency loss function l consistency The image segmentation model can be optimized so that it can generate more stable segmentation results. Therefore, in the training process of the image segmentation model, combining the self-supervised task of judging segmentation consistency with the main task of multi-organ segmentation can improve the encoding and decoding robustness of the image segmentation model and its generalization ability to unknown data sets.

[0077] See also Figure 4 , Figure 4 It is a flowchart of an embodiment of the image segmentation method of the present application.

[0078] Specifically, the following steps may be included:

[0079] Step S41: Acquire a medical image to be segmented.

[0080] Step S42: using the first image segmentation model and / or the second image segmentation model to perform segmentation processing on the medical image to be segmented, to obtain a segmentation result corresponding to the medical image to be segmented.

[0081] The first image segmentation model and the second image segmentation model are obtained by using the above-mentioned image segmentation model training method.

[0082] The image segmentation method of the present application can be used by imaging doctors to observe the three-dimensional morphology of various abdominal organs when assisting in diagnosis and reading films, for example, the smoothness of the three-dimensional mask of the liver surface can be observed; it can also be used to automatically calculate the gray value relationship between various organs through the three-dimensional segmentation results corresponding to the medical image to be segmented, for example, in the plain scan period, the ratio of liver parenchyma to spleen parenchyma is instructive for the evaluation of the degree of fatty liver; in addition, it can also be used for three-dimensional visualization of abdominal organs in surgical planning assistance systems.

[0083] See also Figure 5 , Figure 5It is a schematic diagram of the framework of an embodiment of a training device for an image segmentation model of the present application. The training device 50 of the image segmentation model includes: a sample acquisition module 500, which is used to acquire medical image samples; wherein the medical image samples include a first medical image set annotated with at least one target object and a second medical image set without annotations; an image segmentation module 502, which is used to perform segmentation processing on the first medical image set using the first image segmentation model to obtain a first segmentation result corresponding to the first medical image set; and, respectively, to perform segmentation processing on the second medical image set using the first image segmentation model and the second image segmentation model to obtain a second segmentation result corresponding to the second medical image set segmented by the first image segmentation model, and a third segmentation result corresponding to the second medical image set segmented by the second image segmentation model; wherein the second image segmentation model is obtained by sliding average of the network parameters of the first image segmentation model; a loss function determination module 504, which is used to determine a first loss function of the first image segmentation model based on the first segmentation result; and, based on the second segmentation result and the third segmentation result, determine a second loss function of the first image segmentation model; a parameter adjustment module 506, which is used to adjust the network parameters of the first image segmentation model using the first loss function and the second loss function, and adjust the network parameters of the second image segmentation model based on the adjusted network parameters of the first image segmentation model.

[0084] The above scheme combines the first labeled medical image set and the second unlabeled medical image set together and trains them together in a semi-supervised manner, so that the image segmentation model can have highly robust target object segmentation results and generalization capabilities for unknown medical images.

[0085] In some embodiments, the sample acquisition module 500 can be specifically used to balance each collected medical image set so that the ratio of medical images of each phase in the first medical image set is the same, and / or the ratio of medical images of each phase in the second medical image set is the same.

[0086] In some embodiments, the image segmentation module 502 can also be used to generate a loss function weight for each target object based on the annotation information of the first medical image set and the volume prior information of each target object; wherein the volume prior information of each target object includes the volume mean of all annotations corresponding to the target object. At this time, the loss function determination module 504 can be used to calculate the first loss function based on the first segmentation result and the loss function weight of each target object.

[0087] In some embodiments, the first loss function includes a three-dimensional segmentation loss function, and the loss function determination module 504 is specifically used to analyze the difference information between the first segmentation result and the annotation information in the first medical image set, and calculate the three-dimensional segmentation loss function of the first image segmentation model according to the loss function weight of each target object.

[0088] In some embodiments, the first loss function also includes a two-dimensional projection boundary loss function, and the loss function determination module 504 is specifically used to project the first segmentation result onto a two-dimensional plane, and obtain the segmentation boundary information of each target object based on the projection result; the boundary loss of each target object on the two-dimensional projection is calculated through the segmentation boundary information to obtain the two-dimensional projection boundary loss function of the first image segmentation model.

[0089] In some embodiments, the image segmentation module 502 can be specifically used to add the first random noise to the second medical image set and input it into the first image segmentation model to obtain the second segmentation result, and add the second random noise to the second medical image set and input it into the second image segmentation model to obtain the third segmentation result. The second loss function includes a segmentation consistency loss function, and the loss function determination module 504 can be specifically used to analyze the difference information between the second segmentation result and the third segmentation result to obtain the segmentation consistency loss function of the first image segmentation model.

[0090] In some embodiments, the parameter adjustment module 506 can be specifically used to transfer the first loss function and the second loss function to the first image segmentation model through a reverse transmission algorithm to adjust the network parameters of the first image segmentation model; and perform sliding averaging on the adjusted network parameters of the first image segmentation model to obtain the adjusted network parameters of the second image segmentation model.

[0091] See also Figure 6 , Figure 6 : is a schematic diagram of a framework of an embodiment of an image segmentation device of the present application. The image segmentation device 60 comprises: an image acquisition module 600, used to acquire a medical image to be segmented; an image segmentation module 602, used to perform segmentation processing on the medical image to be segmented using a first image segmentation model and / or a second image segmentation model to obtain a segmentation result corresponding to the medical image to be segmented; wherein the first image segmentation model and the second image segmentation model are obtained using the above-mentioned image segmentation model training method.

[0092] In the above scheme, since the first image segmentation model and the second image segmentation model are obtained by combining the labeled first medical image set and the unlabeled second medical image set and jointly training them in a semi-supervised manner, the first image segmentation model and the second image segmentation model both have highly robust target object segmentation results and generalization capabilities for unknown medical images. Therefore, by using the first image segmentation model and / or the second image segmentation model to perform segmentation processing on the medical image to be segmented, an accurate segmentation result corresponding to the medical image to be segmented can be obtained.

[0093] See also Figure 7 , Figure 7 : is a schematic diagram of a framework of an embodiment of an electronic device of the present application. The electronic device 70 includes a memory 71 and a processor 72 coupled to each other, and the processor 72 is used to execute program instructions stored in the memory 71 to implement the steps of any of the above-mentioned image segmentation model training method embodiments, or the steps of any of the above-mentioned image segmentation method embodiments. In a specific implementation scenario, the electronic device 70 may include but is not limited to: a microcomputer, a server.

[0094] Specifically, the processor 72 is used to control itself and the memory 71 to implement the steps of the training method embodiment of any of the above-mentioned image segmentation models, or the steps of any of the above-mentioned image segmentation method embodiments. The processor 72 can also be called a CPU (Central Processing Unit). The processor 72 may be an integrated circuit chip with signal processing capabilities. The processor 72 can also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field-programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 72 can be implemented by an integrated circuit chip.

[0095] In the above scheme, the processor combines the first labeled medical image set and the second unlabeled medical image set together and performs joint training in a semi-supervised manner, so that the image segmentation model can have highly robust target object segmentation results and generalization capabilities for unknown medical images.

[0096] See also Figure 8 , Figure 8The computer-readable storage medium 80 stores program instructions 800 that can be executed by a processor, and the program instructions 800 are used to implement the steps of any of the above-mentioned training method embodiments of the image segmentation model, or the steps of any of the above-mentioned image segmentation method embodiments.

[0097] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0098] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0099] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0100] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to perform all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

Claims

1. A training method for an image segmentation model, characterized in that: Applied to a first image segmentation model and a second image segmentation model, the second image segmentation model is obtained by sliding average of network parameters of the first image segmentation model; the training method comprises: Acquire medical image samples; wherein the medical image samples include a first medical image set annotated with at least one target object and a second medical image set without annotations; Using the first image segmentation model to perform segmentation processing on the first medical image set to obtain a first segmentation result corresponding to the first medical image set; Based on the annotation information of the first medical image set and the volume prior information of each target object, generating a loss function weight for each target object; wherein the volume prior information of the target object includes the volume mean of all annotations corresponding to the target object; Calculate a first loss function of the first image segmentation model based on the first segmentation result and the loss function weight of each target object; Using the first image segmentation model and the second image segmentation model to perform segmentation processing on the second medical image set respectively, to obtain a second segmentation result corresponding to the second medical image set segmented by the first image segmentation model, and a third segmentation result corresponding to the second medical image set segmented by the second image segmentation model; Determining a second loss function of the first image segmentation model based on the second segmentation result and the third segmentation result; The first loss function and the second loss function are used to adjust the network parameters of the first image segmentation model, and the network parameters of the second image segmentation model are adjusted based on the adjusted network parameters of the first image segmentation model.

2. The image segmentation model training method according to claim 1, characterized in that: The obtaining of medical image samples comprises: A balancing process is performed on each collected medical image set so that the ratio of medical images of each phase in the first medical image set is the same, and / or the ratio of medical images of each phase in the second medical image set is the same.

3. The training method of the image segmentation model according to claim 1, characterized in that: The first loss function includes a three-dimensional segmentation loss function; the first loss function of the first image segmentation model is calculated based on the first segmentation result and the loss function weight of each target object, including: The difference information between the first segmentation result and the annotation information in the first medical image set is analyzed, and the three-dimensional segmentation loss function of the first image segmentation model is calculated according to the loss function weight of each target object.

4. The image segmentation model training method according to claim 3, characterized in that: The first loss function further includes a two-dimensional projection boundary loss function; the first loss function of the first image segmentation model is calculated based on the first segmentation result and the loss function weight of each target object, and further includes: Projecting the first segmentation result onto a two-dimensional plane, and obtaining segmentation boundary information of each target object according to the projection result; The boundary loss of each target object on the two-dimensional projection is calculated using the segmentation boundary information to obtain a two-dimensional projection boundary loss function of the first image segmentation model.

5. The method for training an image segmentation model according to claim 1, characterized in that: The step of using the first image segmentation model and the second image segmentation model to segment the second medical image set respectively to obtain a second segmentation result corresponding to the second medical image set segmented by the first image segmentation model, and a third segmentation result corresponding to the second medical image set segmented by the second image segmentation model, comprises: Adding first random noise to the second medical image set and inputting the result into the first image segmentation model to obtain the second segmentation result, and adding second random noise to the second medical image set and inputting the result into the second image segmentation model to obtain the third segmentation result; The second loss function includes a segmentation consistency loss function; and determining the second loss function of the first image segmentation model based on the second segmentation result and the third segmentation result includes: The difference information between the second segmentation result and the third segmentation result is analyzed to obtain a segmentation consistency loss function of the first image segmentation model.

6. The method for training an image segmentation model according to claim 1, characterized in that: The method of adjusting the network parameters of the first image segmentation model by using the first loss function and the second loss function, and adjusting the network parameters of the second image segmentation model based on the adjusted network parameters of the first image segmentation model comprises: Transmitting the first loss function and the second loss function to the first image segmentation model through a reverse transmission algorithm to adjust network parameters of the first image segmentation model; The adjusted network parameters of the first image segmentation model are sliding averaged to obtain the adjusted network parameters of the second image segmentation model.

7. An image segmentation method, characterized in that: The image segmentation method comprises: Acquire a medical image to be segmented; Using the first image segmentation model and / or the second image segmentation model to perform segmentation processing on the medical image to be segmented, to obtain a segmentation result corresponding to the medical image to be segmented; Wherein, the first image segmentation model and the second image segmentation model are obtained by using the image segmentation model training method described in any one of claims 1 to 6.

8. A training device for an image segmentation model, characterized in that: The training device comprises: A sample acquisition module, used to acquire medical image samples; wherein the medical image samples include a first medical image set annotated with at least one target object and a second medical image set without annotations; An image segmentation module, used to perform segmentation processing on the first medical image set using a first image segmentation model to obtain a first segmentation result corresponding to the first medical image set; and, based on the annotation information of the first medical image set and the volume prior information of each target object, generate a loss function weight for each target object; wherein the volume prior information of the target object includes the volume mean of all annotations corresponding to the target object; and, respectively, perform segmentation processing on the second medical image set using the first image segmentation model and the second image segmentation model to obtain a second segmentation result corresponding to the second medical image set segmented by the first image segmentation model, and a third segmentation result corresponding to the second medical image set segmented by the second image segmentation model; wherein the second image segmentation model is obtained by sliding average of the network parameters of the first image segmentation model; a loss function determination module, configured to calculate a first loss function of the first image segmentation model based on the first segmentation result and the loss function weight of each target object; and determine a second loss function of the first image segmentation model based on the second segmentation result and the third segmentation result; A parameter adjustment module is used to adjust the network parameters of the first image segmentation model using the first loss function and the second loss function, and adjust the network parameters of the second image segmentation model based on the adjusted network parameters of the first image segmentation model.

9. An image segmentation device, characterized in that: The image segmentation device comprises: An image acquisition module, used for acquiring the medical image to be segmented; An image segmentation module, used to perform segmentation processing on the medical image to be segmented using the first image segmentation model and / or the second image segmentation model to obtain a segmentation result corresponding to the medical image to be segmented; Wherein, the first image segmentation model and the second image segmentation model are obtained by using the image segmentation model training method described in any one of claims 1 to 6.

10. An electronic device, characterized in that: It comprises a memory and a processor coupled to each other, wherein the processor is used to execute program instructions stored in the memory to implement the training method of the image segmentation model described in any one of claims 1 to 6, or the image segmentation method described in claim 7.

11. A computer-readable storage medium having program instructions stored thereon, characterized in that: When the program instructions are executed by a processor, the training method of the image segmentation model described in any one of claims 1 to 6, or the image segmentation method described in claim 7 is implemented.

Images