A multi-modal tissue segmentation method and surgical navigation system

Abstract

The application provides a multi-modal tissue segmentation method and a surgical navigation system, and the method comprises the following steps: inputting three-dimensional medical images of multiple modes into corresponding deep learning network models respectively for segmentation to obtain corresponding three-dimensional segmentation results; registering the three-dimensional medical images of each mode to obtain a registration relationship; and according to the registration relationship, combining the three-dimensional medical images of each mode to weight and fuse the three-dimensional segmentation results of each mode according to preset weights of each type of tissue. The application trains a segmentation model for each mode of medical image, each deep learning model has stronger segmentation capability for the medical image of the corresponding mode, and has higher segmentation precision; different types of medical images are respectively designed with preset weights (mode weights) of each type of tissue, and the segmentation results are fused according to the mode weights in the fusion process, so that the tissue structure characteristics are more fully reflected, and the segmentation precision is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of medical image processing, and in particular to a multi-modal tissue segmentation method and a surgical navigation system. BACKGROUND

[0002] Currently, medical image segmentation is usually performed on medical images of a specified type to segment specific tissue structures for analyzing a disease condition and guiding a specific surgery. However, the segmentation accuracy of a single type (modality) of medical image is poor.

[0003] To this end, there are medical image segmentations based on multiple modalities in the prior art.

[0004] One approach is to fuse medical images of multiple modalities to obtain fused medical images, and then input the fused medical images into a pre-trained multi-modal segmentation model to segment tissue structures. This approach has the following defects: Because tissue structures are diverse and medical image imaging principles are different, it is difficult for designers to find a suitable fusion algorithm to retain the features of multiple modalities in the fused medical images, which limits the quality of the fused images and accordingly limits the segmentation accuracy.

[0005] Another approach is to train a model for each modality of medical image, segment medical images of the corresponding modality, and then superimpose and display the segmentation results of various modalities in the same coordinate system. This approach has the following defects: The superimposed display effect is poor, especially when the segmentation results include multiple types of tissue structures, and the superimposed display effect at the junctions of the various types of tissue structures is poor.

[0006] To solve or at least partially solve the above-mentioned defects, the present application provides a multi-modal tissue segmentation method and a surgical navigation system. SUMMARY

[0007] The present application provides a multi-modal tissue segmentation method and a surgical navigation system to solve the defects of poor image fusion quality and fusion results that do not meet user needs in the prior art.

[0008] The present application provides a multi-modal tissue segmentation method, which comprises:

[0009] inputting three-dimensional medical images of multiple modalities into corresponding deep learning network models for segmentation to obtain corresponding three-dimensional segmentation results;

[0010] registering the three-dimensional medical images of each modality to obtain a registration relationship;

[0011] combining the three-dimensional medical images of each modality and weighting the three-dimensional segmentation results of each modality according to the registration relationship and a preset weight of each type of tissue.

[0012] Optionally, the weighting and fusing of the three-dimensional segmentation results of the modalities according to the preset weight of each tissue type and the registration relationship comprises:

[0013] Further, the weighting and fusing of the three-dimensional segmentation results of the modalities according to the preset weight of each tissue type and the registration relationship further comprises:

[0014] For each tissue type, a set of medical images that can be currently used for segmenting the tissue type is determined, and the preset weight of each modality medical image in the set of medical images for the tissue type is obtained;

[0015] The preset weight of each modality medical image in the set of medical images for the tissue type is obtained.

[0016] The weighted probability of the fusion voxel belonging to the tissue type is compared with a preset probability threshold of the tissue type to determine whether the fusion voxel belongs to the tissue type.

[0017] Optionally, the weighting and fusing of the three-dimensional segmentation results of the modalities according to the preset weight of each tissue type and the registration relationship comprises:

[0018] The weighted probability of the fusion voxel belonging to each tissue type is determined by modality weighting according to the registration relationship.

[0019] The tissue type corresponding to the maximum weighted probability is determined as the fusion label of the voxel.

[0020] Further, the weighted probability of the fusion voxel belonging to each tissue type is determined by modality weighting, comprising:

[0021] A set of currently available medical images is determined, and the preset weight of each modality medical image in the set of medical images for each tissue type is obtained.

[0022] The modality weight of each modality medical image in the set of medical images for the tissue type is obtained.

[0023] Optionally, the weighting and fusing the three-dimensional segmentation results of each modality according to the preset weight of each type of tissue and the registration relationship further comprises: converting the three-dimensional segmentation results of each modality to the same coordinate system according to the registration relationship, so as to view the segmentation results of each modality.

[0024] Optionally, the registration of the three-dimensional medical images of each modality to obtain the registration relationship comprises:

[0025] unifying the sizes of the first modality medical image and the second modality medical image;

[0026] registering the first modality medical image and the second modality medical image based on the maximum mutual information to obtain the registration relationship therebetween.

[0027] Further, the registration of the three-dimensional medical images of each modality to obtain the registration relationship further comprises: guiding the registration process of the first modality medical image and the second modality medical image by using the segmentation structure of the first modality medical image and the segmentation result of the second modality medical image.

[0028] The application also provides a computer program product, which comprises computer executable instructions, the instructions being used to implement the multi-modality tissue segmentation method according to any one of the above when executed.

[0029] The application also provides a surgical navigation system, which comprises a host computer and a display device, wherein the host computer stores executable instructions, and the executable instructions are used to implement the multi-modality tissue segmentation method according to any one of the above when executed by the host computer.

[0030] The application also provides a non-transitory computer readable storage medium, which stores a computer program, and the computer program is used to implement the multi-modality tissue segmentation method according to any one of the above when executed by a processor.

[0031] The multi-modality tissue segmentation method and the surgical navigation system provided by the application have at least the following beneficial effects:

[0032] 1. The segmentation model is trained for each modality medical image, and each deep learning model has stronger segmentation ability for the corresponding modality medical image and higher segmentation accuracy; different types of medical images are designed with respective preset weights (modality weights) for each type of tissue, and the segmentation results are fused according to the modality weights in the fusion process, so that the tissue structure features are more fully reflected and the segmentation accuracy is improved.

[0033] 2. In a fusion mode, the segmentation results of each modality are fused by modality weighting for each type of tissue (i.e., the fusion results can be managed by tissue type), and the doctor can switch to view different types of tissue structures, facilitating the viewing of the structure and morphology of specific tissues (such as observing the morphology and size of a lesion).

[0034] 3. In another fusion mode, the weighted probability of each type of tissue structure to which the fusion voxel belongs can also be determined, and the tissue type to which the fusion voxel belongs can be determined by comparing the probabilities (i.e., voxel-level fusion can be achieved, and each type of tissue can be displayed), which can reasonably fuse and display the segmentation results of different modalities, especially improving the accuracy of the junction of different types of tissues, facilitating the viewing of the spatial position relationship of different tissues, and facilitating the discussion and planning of surgical plans.

[0035] 4. In another fusion mode, the fusion results can also be managed by modality, i.e., the doctor can view each type of tissue segmented by each modality, and the segmentation results of each modality can also be displayed with a certain degree of transparency.

[0036] 5. In a preferred scheme, various combinations of the above three fusion modes are included, and the doctor can flexibly select and view the fusion results under the corresponding fusion mode, facilitating the viewing and planning of tissues and surgery. BRIEF DESCRIPTION OF DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings described below are some embodiments of the present application, and those skilled in the art can obtain other drawings according to these drawings without creative labor.

[0038] Figure 1 is a flowchart of the multi-modality tissue segmentation method provided by the present application;

[0039] Figure 2 is a structural schematic diagram of the surgical navigation system provided by the present application. DETAILED DESCRIPTION

[0040] In order to make the objects, technical solutions and advantages of the present application clearer, the technical solutions in the present application will be described clearly and completely below in combination with the drawings in the present application. Obviously, the described embodiments are some embodiments of the present application, rather than all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0041] The following will be described in combination with Figures 1-2The present invention describes a multimodal tissue segmentation method and a surgical navigation system. Figure 1 This is a flowchart illustrating a multimodal tissue segmentation method provided by the present invention, as shown below. Figure 1 As shown, the method includes:

[0042] S1. Input the three-dimensional medical images of various modalities into the corresponding deep learning network models for segmentation to obtain the corresponding three-dimensional segmentation results;

[0043] Specifically, deep learning models for segmenting various tissues are pre-trained for each modality of medical images. Examples of medical images include CT, ultrasound, DSA (digital subtraction angiography), MRI (magnetic resonance imaging), MRA (magnetic resonance angiography), CTA (computational CT angiography), PET (positron emission tomography), SPECT (single-photon emission computed tomography), and DTI (diffusion tensor imaging).

[0044] Deep learning models are machine learning algorithms based on artificial neural networks. They perform computation and learning through a multi-layered network structure (including input, hidden, and output layers), extracting shallow, mid-level, and high-level features from the original input layer by layer, and completing the transformation and mapping of the task objective (such as image recognition, speech recognition, and natural language processing). In this embodiment, a deep learning model is pre-trained for each modality of medical images, making the trained model more targeted and achieving higher segmentation accuracy for the corresponding modality of medical images. The model can be a binary classification model, for example, a model segmenting "spine" and "background" in a CT image, and a model segmenting "blood vessels" and "background" in a DSA image. The model can also be a multi-class classification model, for example, a model segmenting "skull," "tumor," and "background" in a CT image, and a model segmenting "blood vessels," "tumor," "brain functional area," and "background" in an MRI image. Of course, some models can be binary classification models, and some models can be multi-class classification models.

[0045] S2. Register the three-dimensional medical images of each modality to obtain the registration relationship;

[0046] Specifically, the 3D medical images of each modality are registered, and the resulting registration relationship indicates the correspondence between voxels in each modality of medical images, facilitating the fusion of segmentation results from each modality. Furthermore, the registration process can use a 3D medical image of one modality as a reference image, and register other 3D medical images with that reference image respectively. The reference image should be selected as the one that best represents the patient's current true condition, such as an MRI image, or an image acquired most recently.

[0047] It can be understood that there is no sequence between the above steps S1 and S2, and the registration can be performed first, and then the medical images of each modality are segmented. The segmentation can also be performed first, and then the medical images of each modality are registered. Further, the segmentation result can also be used to guide the registration process to improve the registration accuracy. For example, the time interval between the two medical image acquisition of the patient is long, and the tissue morphology has changed, and it is difficult to register based on the medical image only. The segmentation result can improve the registration accuracy. The registration here can be rigid registration or non-rigid registration.

[0048] S3. According to the registration relationship, the three-dimensional segmentation results of each modality are weighted and fused by combining the preset weights of each modality for each type of tissue.

[0049] Specifically, the three-dimensional medical images of each modality respectively preset their own weights (i.e., modality weights) for each type of tissue. For example, the medical images of each modality include MRI, DSA, and ultrasonic imaging. These modalities of medical images can all segment out "blood vessels", "tumors", and "backgrounds". The above medical images of each modality respectively preset their own weights for "blood vessels" and other weights for "tumors". When performing fusion, the preset weights of each modality for each type of tissue are combined for fusion, so that the fusion result is more accurate.

[0050] The above weights are flexibly set according to the characteristics of the medical images. For example, DSA has a greater weight for "blood vessels" than MRI and ultrasonic imaging, and MRI has a greater weight for "tumors" than DSA and ultrasonic imaging. By presetting the weights, the characteristics can be retained to a greater extent. The above preset weights can also be compatible with the case of missing modalities. For example, DSA examination takes a long time, and only MRI and ultrasonic images are collected before treatment. At this time, only the segmentation results of MRI and ultrasonic images can be fused.

[0051] The embodiment trains a deep learning model for each modality of medical image, each model has stronger segmentation ability for the corresponding modality of medical image, and has higher segmentation accuracy. Different types of medical images respectively design their own preset weights (modality weights) for each type of tissue. In the fusion process, the segmentation results are fused according to the modality weights, which more greatly reflects the tissue structure characteristics and improves the multi-modality segmentation accuracy. The above fusion method can also be compatible with the case of missing modalities, which expands the application range.

[0052] Based on the previous embodiment, in one embodiment, the multiple modalities of three-dimensional medical images in step S1 include cross-modality images generated by a large model. Specifically, the large model is a cross-modality image generation model that has the ability to "generate another modality medical image according to an existing modality medical image" after being pre-trained. For example, in one scenario, the T1 image of the patient is collected, the T1 image is input into the pre-trained cross-modality image generation model, and the model generates the T2 image of the patient. Thereafter, the "T1 image, T2 image" can be used for subsequent multi-modality tissue segmentation and fusion. For another example, in another scenario, the MRI image and CT image of the patient are collected, and the CTA image is missing. The MRI image and CT image of the patient are input into the trained cross-modality image generation model, and the model generates the CTA image of the patient. Thereafter, the "MRI image, CT image, CTA image" can be used for subsequent multi-modality tissue segmentation and fusion, and the tissue structure features are more accurately and comprehensively presented in the fused image to provide information reference for doctors.

[0053] Preferably, the above-mentioned cross-modality image generation model converts between different modalities collected by the same type of device to enhance the quality of the generated image. For example, the T1 image and the T2 image are both images collected by a magnetic resonance device, and the cross-modality image generation model converts between T1 and T2. For another example, EBT and PET-CT are both images collected by a CT device, and the cross-modality image generation model converts between EBT and PET-CT.

[0054] Based on any of the embodiments, in one embodiment, S3 includes: weighting and fusing the three-dimensional segmentation results of each modality respectively according to the tissue categories.

[0055] Specifically, the present embodiment provides a fusion method for weighting and fusing the segmentation results of each modality according to the tissue categories. For example, the three-dimensional medical images of the patient include CT, MRI, DSA, PET, and ultrasound, and the above-mentioned images can be segmented into "tumors", and the MRI, DSA, and ultrasound can be further segmented into "blood vessels". When fusing, the "tumor" segmentation results of each modality are weighted and fused according to the preset weights of the CT, MRI, DSA, PET, and ultrasound images for "tumors", and the "blood vessel" segmentation results of each modality are weighted and fused according to the preset weights of the MRI, DSA, and ultrasound images for "blood vessels".

[0056] It can be understood that the modality weight combination of "tumors" is different from the modality weight combination of "blood vessels", and the weighting and fusing processes are not the same. By presetting the weights, the influence factors of different modality medical images on different tissue types are reasonably adjusted, so that the fusion result is more reasonable, accurate, and comprehensive.

[0057] The embodiment provides a fusion mode, which fuses segmentation results of each modality in a mode weighting manner for each type of tissue (that is, the fusion results can be managed respectively according to the tissue types), and doctors can switch to view different types of tissue structures, thereby facilitating viewing of the structure and morphology of specific tissues (such as observation of lesion morphology and size).

[0058] Based on any one of the embodiments, in one embodiment, S3 further includes:

[0059] For each type of tissue, determine a set of medical images that can be currently used for segmenting the type of tissue, and obtain preset weights of each modality medical image in the set of medical images for the tissue;

[0060] According to the preset weights of each modality medical image in the set of medical images for the type of tissue, and based on the registration relationship, weighted fusion is performed on the three-dimensional segmentation results of each modality for the type of tissue, to obtain a weighted probability that a fusion voxel belongs to the type of tissue;

[0061] The weighted probability that the fusion voxel belongs to the type of tissue is compared with a preset probability threshold of the tissue, to determine whether the fusion voxel belongs to the type of tissue.

[0062] Specifically, in actual application, only part of the medical images of a patient in a modality can be obtained. The method of the present application can be compatible with the case where a modality is missing, and the medical images in part of the modalities can still be segmented and fused.

[0063] For example, Table 1 below shows one example of preset weights of each type of medical image for each type of tissue:

[0064]

[0065] Table 1

[0066] CT, MRI, DSA and ultrasound images of a patient are collected, and the PET modality is missing. The fusion process is as follows:

[0067] For the “blood vessel” tissue, the set of medical images that can currently segment the “blood vessel” is determined from the above-mentioned medical images in multiple modalities as MRI, DSA and ultrasound, and the corresponding preset weights are 3, 6 and 1 respectively.

[0068] The segmented "blood vessels" in the MRI, DSA and ultrasound images are fused according to the above-mentioned weights. For example, according to the registration relationship, it is determined that the segmentation result of MRI at a certain voxel is "blood vessel", the segmentation result of DSA is "blood vessel", and the segmentation result of ultrasound is "non-blood vessel", and the weighted probability of this fusion voxel is (1x3+1x6+0x1) / 10=0.9, it is determined that 0.9 is greater than the preset probability threshold 0.5 of "blood vessel", the fusion voxel belongs to "blood vessel", and other voxels can be sequentially determined whether they belong to "blood vessel" by referring to this process, so as to obtain the fusion result of "blood vessel".

[0069] For "tumor" tissue, it is determined that the current mode that can segment "tumor" is CT, MRI, DSA and ultrasound, and the corresponding preset weights are 1, 3, 4 and 1 respectively.

[0070] The segmented "tumor" in the CT, MRI, DSA and ultrasound images is fused according to the above-mentioned weights. For example, according to the registration relationship, it is determined that the segmentation result of CT at a certain voxel is "tumor", the segmentation result of MRI is "tumor", the segmentation result of DSA is "tumor", and the segmentation result of ultrasound is "non-tumor", and the weighted probability of this fusion voxel is (1x1+0x3+1x4+0x1) / 9=0.55, it is determined that 0.66 is greater than the preset probability threshold 0.6 of "tumor", the fusion voxel belongs to "non-tumor", and other voxels can be sequentially determined whether they belong to "tumor" by referring to this process, so as to obtain the fusion result of "tumor". Similarly, other tissues such as "skull" can also be fused by weighting.

[0071] The embodiment realizes the fusion of various types of tissues, and users can conveniently view different types of tissue structures. In the fusion process, the segmentation results of each modality are fused by modality weighting for each type of tissue, the fusion results can be managed according to the type of tissue, and doctors can conveniently switch to view different types of tissue structures, for example, observe the morphology and size of a specific lesion tissue.

[0072] Based on any one of the embodiments, in one embodiment, S3 comprises:

[0073] According to the registration relationship, the voxels are fused, and the weighted probability of the fusion voxel belonging to each type of tissue is determined by modality weighting;

[0074] The type of tissue corresponding to the maximum weighted probability is determined as the fusion label of the voxel.

[0075] Specifically, the embodiment provides another weighted fusion manner, realizes voxel-level fusion, and observes various types of tissues in a three-dimensional image. In the fusion process, the weighted probability of a fusion voxel belonging to each type of tissue is calculated by combining the preset weight, the greater the weighted probability, the higher the possibility that the fusion voxel belongs to the corresponding tissue type, and the tissue type corresponding to the maximum weighted probability is determined as the fusion label of the voxel. Further, different types of tissues in the fusion image are rendered in different colors, facilitating viewing.

[0076] The embodiment provides still another fusion manner, which reasonably adjusts the influence factors of different modal medical images on different tissue types by the preset weight, so that the calculation of the weighted probability is more reasonable and accurate, the label of the fused voxel is determined by comparing the weighted probabilities of various tissue types, so that the fusion result is more reasonable, and the fusion accuracy at the junction of different types of tissues is improved. The tissue structure that can be displayed in the fusion image facilitates doctors to view the spatial position relationship of different tissues and facilitates discussion and planning of a surgical plan.

[0077] Based on any embodiment, in one embodiment, the weighted probability of a fusion voxel belonging to each type of tissue is determined by modal weighting, comprising:

[0078] A set of currently available medical images is determined, and the preset weight of each modality medical image in the set of medical images on each type of tissue is obtained;

[0079] The weighted probability of a fusion voxel belonging to each type of tissue is determined by modal weighting, comprising:

[0080] Specifically, in actual application, only part of the modal medical images of a patient can be obtained. The method of the present application can be compatible with the lack of modalities, and the medical images of part of the modalities can still be segmented and fused.

[0081] For example, Table 2 below shows another example of the preset weight of each type of medical image on each type of tissue:

[0082]

[0083] Table 2

[0084] The CT, DSA, PET, and ultrasound images of a patient are collected, and the MRI modality is missing. The fusion process is as follows:

[0085] The preset weight of the CT, DSA, PET, and ultrasound image modalities on the "skull", "blood vessels", "tumor", and "brain function area" is obtained from Table 2, respectively.

[0086]

[0087]

[0088] Table 3

[0089] Table 3 shows the segmentation results of a voxel under each modality image. According to the preset weight obtained, the segmentation results are weighted and calculated respectively to obtain the weighted probability of the voxel belonging to each type of tissue:

[0090] The weighted probability of belonging to the skull: (0x8+0x2) / 10=0;

[0091] The weighted probability of belonging to the blood vessel: (1x5+1x1) / 6=1;

[0092] The weighted probability of belonging to the tumor: (0x1+0x1+0x4+0x0.5) / 6.5=0;

[0093] The weighted probability of belonging to the brain function area: (0x2) / 2=0;

[0094] The weighted probability of belonging to the background: (1x1+0x1+1x1+0x1) / 4=0.5.

[0095] By comparison, it is determined that the weighted probability of the voxel belonging to the blood vessel is the largest, and the fusion result of the voxel is determined as "blood vessel". Referring to this process, the fusion results of other voxels are determined respectively, and finally the fusion results of the segmentation results of each modality are obtained.

[0096] In this embodiment, the preset weights assigned to the medical images of different modalities for different types of tissues are different, and the influence factors of different modalities on different types of tissues are reasonably adjusted by the preset weights, so that the calculation of the weighted probability is more reasonable, the fusion result is more accurate, and the fusion accuracy at the junction of the tissues is improved.

[0097] Based on any one of the embodiments, in one embodiment, S3 further includes: converting the three-dimensional segmentation results of each modality to the same coordinate system according to the registration relationship, so as to view the segmentation results of each modality.

[0098] This embodiment further provides another fusion manner based on the foregoing fusion manner. Specifically, the segmentation results of each modality are converted to the same coordinate system according to the registration relationship, and the doctor can switch to view the segmentation results of each type of tissue under each modality. It can be understood that the types of tissues that can be segmented by medical images of different modalities are different, and the segmentation accuracy for different tissues is also different. The initial segmentation results of each modality provide more basic information, which facilitates the doctor to further analyze the tissue structure.

[0099] Based on any of the embodiments, in one embodiment, S2 comprises:

[0100] The first modality medical image and the second modality medical image are unified in size; the first modality medical image and the second modality medical image are registered based on the maximum mutual information, to obtain a registration relationship therebetween.

[0101] The same size process is, for example, unifying the medical images of each modality to the size of a certain modality medical image, and for example, unifying the medical images of each modality to the same standard size. The same modality medical image size can improve the registration accuracy and reduce the registration calculation amount. In the registration process, the mutual information can be used for registration based on the maximum mutual information. The mutual information can be used to describe the similarity between two images, so that the registration parameter with the maximum mutual information is the final registration relationship. The mutual information can be calculated based on the gray histogram or based on the probability.

[0102] The embodiment can guide the fusion process and achieve high-precision fusion by unifying the medical images of each modality in size and obtaining the registration relationship based on the maximum mutual information.

[0103] Based on any of the embodiments, in one embodiment, S2 further comprises: using the segmentation structure of the first modality medical image and the segmentation result of the second modality medical image to guide the registration process of the first modality medical image and the second modality medical image.

[0104] Specifically, in the embodiment, step S1 is executed before step S2. Since the original images of different modalities have large differences and the registration accuracy is low, after the image is segmented in step S1, the segmentation result is used to guide the registration process of step S2, for example, the edge features of the segmentation result are extracted and added to the original image to guide the registration process, so that the registration accuracy is higher and the registered image is more consistent with the true situation of the patient, which is beneficial to improve the fusion accuracy.

[0105] The application also provides a surgical navigation system. The surgical navigation system described below can be referred to with the multi-modality tissue segmentation method described above.

[0106] Figure 2 is a structural schematic diagram of a surgical navigation system provided by the application, as Figure 2 shown, the surgical navigation system comprises a host computer 21 and a display device 22.

[0107] The host computer 100 stores executable instructions. When the executable instructions are executed by the host computer 100, the multi-modality tissue segmentation method of any of the preceding embodiments is realized.

[0108] The display device 22 is used to display medical images, interactive interfaces, fusion results, etc., for the convenience of doctors to view.

[0109] The embodiment can provide information support for preoperative surgical plan by multi-modal tissue segmentation, and further present the patient three-dimensional model / surgical plan / surgical instrument and the like in the display device 22 during the surgical navigation process to guide the surgical process.

[0110] The present application also provides a computer program product comprising computer executable instructions for implementing the multi-modal tissue segmentation method of any of the preceding embodiments when executed.

[0111] The present application also provides a non-transitory computer readable storage medium having stored thereon a computer program for implementing the multi-modal tissue segmentation method of any of the preceding embodiments when executed by a processor.

[0112] The system embodiments described above are merely illustrative, wherein the units illustrated as separate components can or can not be physically separated, and the components illustrated as units can or can not be physical units, i.e., can be located in one place, or can be distributed on multiple network units. Part or all of the modules can be selected to achieve the purpose of the embodiment according to actual needs. Those skilled in the art can understand and implement without creative labor.

[0113] From the above description of the embodiments, those skilled in the art can clearly understand that the embodiments can be implemented by means of software plus necessary universal hardware platforms, and of course can also be implemented by hardware. Based on such understanding, the above technical solutions, essentially or in other words, the part that contributes to the prior art, can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes a number of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods of the embodiments or some parts of the embodiments.

[0114] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement to some technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A multimodal tissue segmentation method, characterized in that, include: Multiple modalities of 3D medical images are input into corresponding deep learning network models for segmentation to obtain corresponding 3D segmentation results; The 3D medical images of each modality are registered to obtain the registration relationship; Based on the registration relationship, the three-dimensional segmentation results of each modality are weighted and fused by combining the preset weights of each type of tissue with the three-dimensional medical images of each modality. The step of weightedly fusing the 3D segmentation results of each modality based on the registration relationship and the preset weights of each type of tissue in the 3D medical images of each modality further includes: For each type of tissue, determine the set of medical images that can be used to segment that type of tissue, and obtain the preset weights of each modality of medical images for that tissue. Based on the preset weights of each modal medical image in the medical image set for this type of tissue, and based on the registration relationship, the three-dimensional segmentation results of each modality of this type of tissue are weighted and fused to obtain the weighted probability that the fused voxel belongs to this type of tissue. The weighted probability of the fusion voxel belonging to this type of tissue is compared with the preset probability threshold of this tissue to determine whether the fusion voxel belongs to this type of tissue.

2. The multimodal tissue segmentation method according to claim 1, characterized in that, The step of weightedly fusing the three-dimensional segmentation results of each modality according to the registration relationship and the preset weights of each type of tissue in the three-dimensional medical images of each modality also includes: fusing and transforming the three-dimensional segmentation results of each modality to the same coordinate system according to the registration relationship, so as to view the segmentation results of each modality.

3. The multimodal tissue segmentation method according to claim 1, characterized in that, The registration of the three-dimensional medical images of each modality to obtain the registration relationship includes: Standardize the size of the first and second modality medical images; Based on maximum mutual information, the first modality medical image and the second modality medical image are registered to obtain the registration relationship between them.

4. The multimodal tissue segmentation method according to claim 3, characterized in that, The process of registering the three-dimensional medical images of each modality to obtain the registration relationship also includes: using the segmentation results of the first modality medical image and the segmentation results of the second modality medical image to guide the registration process between the first modality medical image and the second modality medical image.

5. A multimodal tissue segmentation method, characterized in that, include: Multiple modalities of 3D medical images are input into corresponding deep learning network models for segmentation to obtain corresponding 3D segmentation results; The 3D medical images of each modality are registered to obtain the registration relationship; Based on the registration relationship, the three-dimensional segmentation results of each modality are weighted and fused by combining the preset weights of each type of tissue with the three-dimensional medical images of each modality. The step of weightedly fusing the 3D segmentation results of each modality based on the registration relationship and the preset weights of each type of tissue in the 3D medical images of each modality includes: According to the registration relationship, voxels are fused one by one, and the weighted probability of the fused voxel belonging to each tissue type is determined by modal weighting. The tissue type corresponding to the highest weighted probability is determined as the fusion label for that voxel; The method of determining the weighted probability of a fusion voxel belonging to each tissue type through modal weighting includes: Determine the currently available set of medical images and obtain the preset weights of each modality of medical images for various tissues; Based on the modal weights of each modal medical image in the medical image set for this type of tissue, and based on the registration relationship, the segmentation results of each modality of the first tissue are weighted and fused to obtain the weighted probability that the fused voxel belongs to the first tissue; wherein, the first tissue is one of multiple tissue types.

6. The multimodal tissue segmentation method according to claim 5, characterized in that, The step of weightedly fusing the three-dimensional segmentation results of each modality according to the registration relationship and the preset weights of each type of tissue in the three-dimensional medical images of each modality also includes: fusing and transforming the three-dimensional segmentation results of each modality to the same coordinate system according to the registration relationship, so as to view the segmentation results of each modality.

7. The multimodal tissue segmentation method according to claim 5, characterized in that, The registration of the three-dimensional medical images of each modality to obtain the registration relationship includes: Standardize the size of the first and second modality medical images; Based on maximum mutual information, the first modality medical image and the second modality medical image are registered to obtain the registration relationship between them.

8. The multimodal tissue segmentation method according to claim 7, characterized in that, The process of registering the three-dimensional medical images of each modality to obtain the registration relationship also includes: using the segmentation results of the first modality medical image and the segmentation results of the second modality medical image to guide the registration process between the first modality medical image and the second modality medical image.

9. A computer program product, said computer program product comprising computer-executable instructions, characterized in that, When executed, the instructions are used to implement the multimodal tissue segmentation method as described in any one of claims 1-4 or 5-8.

10. A surgical navigation system, comprising a host computer and a display device, characterized in that, The host computer stores executable instructions, which, when executed by the host computer, implement the multimodal tissue segmentation method as described in any one of claims 1-4 or 5-8.

Images