Multi-modal tissue segmentation method and surgical navigation system

Abstract

The invention provides a multi-modal tissue segmentation method and a surgical navigation system, and the method comprises the steps: inputting a multi-modal three-dimensional medical image into a corresponding deep learning network model for segmentation, and obtaining a corresponding three-dimensional segmentation result; registering the three-dimensional medical image of each modal to obtain a registration relation; and according to the registration relationship, combining the preset weight of the three-dimensional medical image of each modal for each type of tissue, and carrying out weighted fusion on the three-dimensional segmentation results of each modal. According to the method, the segmentation model is trained for the medical image of each modal, and each deep learning model has higher segmentation capability for the medical image of the corresponding modal and higher segmentation precision; different types of medical images respectively design respective preset weights (modal weights) for each type of tissues, and the segmentation results are fused according to the modal weights in the fusion process, so that the tissue structure features are reflected to a greater extent, and the segmentation precision is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image processing, and in particular to a multimodal tissue segmentation method and a surgical navigation system. Background Art

[0002] At present, the common practice of medical image segmentation is to segment medical images of the type specified by the doctor in order to segment out specific tissue structures for analyzing the condition and guiding specific surgery. However, the accuracy of single-type (modality) medical image segmentation is poor.

[0003] In this regard, there is multi-modality-based medical image segmentation in the prior art.

[0004] One approach is to fuse medical images of multiple modalities to obtain a fused medical image, and then input the fused medical image into a pre-trained multimodal segmentation model to segment the tissue structure. The drawback of this approach is that due to the variety of tissue structures and the different imaging principles of medical images, it is difficult for designers to find a suitable fusion algorithm to retain the features of multiple modalities in the fused medical image, which limits the quality of the fused image and, accordingly, the accuracy of the segmentation.

[0005] Another approach is to train models for different modalities of medical images, segment the corresponding modalities of medical images, and then overlay the segmentation results of various modalities and display them in the same coordinate system. The disadvantage of this approach is that the overlay display effect is poor, especially when the segmentation results include multiple types of tissue structures, the overlay display effect at the junction of various tissue structures is poor.

[0006] In order to solve or at least partially solve the above-mentioned defects, the present invention provides a multimodal tissue segmentation method and a surgical navigation system. Summary of the invention

[0007] The present invention provides a multimodal tissue segmentation method and a surgical navigation system, which are used to solve the defects in the prior art that the image fusion quality is poor and the fusion results do not meet the user's use requirements.

[0008] The present invention provides a multimodal tissue segmentation method, comprising:

[0009] Input three-dimensional medical images of various modalities into corresponding deep learning network models for segmentation, and obtain corresponding three-dimensional segmentation results;

[0010] Register the three-dimensional medical images of each modality to obtain the registration relationship;

[0011] According to the registration relationship, the three-dimensional segmentation results of each modality are weighted and fused in combination with the preset weights of each type of tissue in the three-dimensional medical images of each modality.

[0012] Optionally, the three-dimensional segmentation results of each modality are weighted and fused according to the registration relationship and in combination with the preset weights of each type of tissue in the three-dimensional medical images of each modality, including: weighted fusion of the three-dimensional segmentation results of each modality according to tissue category.

[0013] Furthermore, the weighted fusion of the three-dimensional segmentation results of each modality based on the registration relationship and the preset weights of each type of tissue in the three-dimensional medical images of each modality further includes:

[0014] For each type of tissue, determine the medical image set that can be used to segment the tissue, and obtain the preset weight of each modality of medical image for the tissue;

[0015] According to the preset weights of each modality of medical images in the medical image set for the type of tissue, and based on the registration relationship, weighted fusion is performed on the three-dimensional segmentation results of each modality of the type of tissue to obtain a weighted probability that the fused voxel belongs to the type of tissue;

[0016] The weighted probability that the fused voxel belongs to the type of tissue is compared with a preset probability threshold of the tissue to determine whether the fused voxel belongs to the type of tissue.

[0017] Optionally, the weighted fusion of the three-dimensional segmentation results of each modality based on the registration relationship and the preset weight of each type of tissue in the three-dimensional medical images of each modality includes:

[0018] fusing each voxel according to the registration relationship, and determining the weighted probability that the fused voxel belongs to each tissue type by means of modality weighting;

[0019] The tissue type corresponding to the maximum weighted probability is determined as the fusion label of the voxel.

[0020] Furthermore, determining the weighted probability that the fused voxel belongs to each tissue type by means of modality weighting includes:

[0021] Determine the currently available medical image set and obtain the preset weights of each modality of medical image for each type of tissue;

[0022] According to the modal weights of each modality of medical images in the medical image set for this type of tissue, and based on the registration relationship, weighted fusion is performed on the segmentation results of each modality of the first tissue to obtain a weighted probability that the fused voxel belongs to the first tissue; wherein the first tissue is one of multiple tissue types.

[0023] Optionally, the weighted fusion of the three-dimensional segmentation results of each modality based on the registration relationship and the preset weights of each type of tissue in the three-dimensional medical images of each modality also includes: according to the registration relationship, the fusion and conversion of the three-dimensional segmentation results of each modality into the same coordinate system so as to view the segmentation results of each modality.

[0024] Optionally, registering the three-dimensional medical images of each modality to obtain a registration relationship includes:

[0025] unifying the size of the first modality medical image and the second modality medical image;

[0026] Based on the maximum mutual information, the first modality medical image and the second modality medical image are registered to obtain a registration relationship between the two.

[0027] Furthermore, the registering of the three-dimensional medical images of each modality to obtain the registration relationship also includes: using the segmentation structure of the first modality medical image and the segmentation result of the second modality medical image to guide the registration process of the first modality medical image and the second modality medical image.

[0028] The present invention also provides a computer program product, comprising computer executable instructions, which are used to implement any of the multimodal tissue segmentation methods described above when executed.

[0029] The present invention also provides a surgical navigation system, comprising: a host and a display device, characterized in that the host stores executable instructions, and when the executable instructions are executed by the host, any of the multimodal tissue segmentation methods described above is implemented.

[0030] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the multimodal tissue segmentation method as described above is implemented.

[0031] The multimodal tissue segmentation method and surgical navigation system provided by the present invention have at least the following beneficial effects:

[0032] 1. A segmentation model was trained for each modality of medical images. Each deep learning model has a stronger ability to segment medical images of the corresponding modality and higher segmentation accuracy. Different types of medical images are designed with their own preset weights (modality weights) for each type of tissue. During the fusion process, the segmentation results are fused according to the modality weights, which reflects the tissue structure characteristics to a greater extent and improves the segmentation accuracy.

[0033] 2. In one fusion method, the segmentation results of each modality are fused for each type of tissue by modality weighting (i.e., the fusion results can be managed separately by tissue type). Doctors can switch to view different types of tissue structures, which is convenient for viewing the structure and morphology of specific tissues (such as observing the morphology and size of lesions).

[0034] 3. In another fusion method, the weighted probability that the fused voxel belongs to each type of tissue structure can be determined separately, and the tissue type to which the fused voxel belongs can be determined by comparing the probabilities (that is, voxel-level fusion can be achieved and various types of tissues can be displayed at the same time). The segmentation results of different modalities can be reasonably fused and displayed, especially the accuracy of the junction of different types of tissues is improved, which makes it convenient to view the spatial position relationship of different tissues and facilitate the discussion and planning of surgical plans.

[0035] 4. In another fusion method, the fusion results can also be managed by modality, that is, the doctor can view the various types of tissues segmented by each modality separately, and the segmentation results of each modality can also be superimposed and displayed with a certain degree of transparency.

[0036] 5. In a preferred solution, various combinations of the above three fusion methods are included, and doctors can flexibly select and view the fusion results under the corresponding fusion method, which is convenient for viewing tissues and planning operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0038] Figure 1 It is a schematic flow chart of the multimodal tissue segmentation method provided by the present invention;

[0039] Figure 2 It is a structural schematic diagram of the surgical navigation system provided by the present invention. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0041] Combine the following Figure 1-Figure 2The multimodal tissue segmentation method and surgical navigation system of the present invention are described. Figure 1 is a flow chart of a multimodal tissue segmentation method provided by the present invention, such as Figure 1 As shown, the method includes:

[0042] S1. Input three-dimensional medical images of multiple modalities into corresponding deep learning network models for segmentation, and obtain corresponding three-dimensional segmentation results;

[0043] Specifically, deep learning models for segmenting various tissues are pre-trained for each modality of medical imaging, such as CT, ultrasound, DSA (digital subtraction angiography), MRI (magnetic resonance imaging), MRA (magnetic resonance angiography), CTA (CT angiography), PET (positron emission tomography), SPECT (single photon emission computed tomography), DTI (diffusion tensor imaging), etc.

[0044] The deep learning model is a machine learning algorithm based on an artificial neural network. It calculates and learns through a multi-layer network structure (including input layer, hidden layer, and output layer), extracts the shallow features, middle features, and high-level features of the original input layer by layer, and completes the transformation mapping of the task objectives (such as completing image recognition, speech recognition, natural language processing, etc.). In this embodiment, a deep learning model is pre-trained for each modality of medical images, so that the trained model is more targeted and the segmentation accuracy of the medical images of the corresponding modality is higher. The model can be a binary classification model, for example, a certain model segments "spine" and "background" for CT images, and a certain model segments "blood vessels" and "background" for DSA images. The model can also be a multi-classification model, for example, a certain model segments "skull", "tumor", and "background" for CT images, and a certain model segments "blood vessels", "tumors", "brain functional areas", and "background" for MRI images. Of course, some models can also be binary classification models and some models can be multi-classification models.

[0045] S2. registering the three-dimensional medical images of each modality to obtain a registration relationship;

[0046] Specifically, the three-dimensional medical images of each modality are registered, and the obtained registration relationship indicates the correspondence between the voxels in the medical images of each modality, which is convenient for fusing the segmentation results of each modality on this basis. Furthermore, the registration process can use the three-dimensional medical image of a certain modality as the reference image, and register the three-dimensional medical images of other modalities with the reference image respectively. The reference image should be selected as the image that best expresses the patient's current true state, such as using an MRI image as the reference image, or, for example, using an image with the most recent acquisition time as the reference image.

[0047] It is understandable that there is no order relationship between the above steps S1 and S2. Registration can be performed first, and then the medical images of each modality can be segmented. Segmentation can also be performed first, and then the medical images of each modality can be registered. Furthermore, the segmentation results can also be used to guide the registration process and improve the registration accuracy. For example, the time interval between the acquisition of two medical images of a patient is long, and the tissue morphology has changed. It is difficult to register based on medical images alone. The segmentation results can be used to improve the registration accuracy. The registration here can be rigid registration or non-rigid registration.

[0048] S3. According to the registration relationship, the 3D segmentation results of each modality are weighted and fused in combination with the preset weights of each type of tissue in the 3D medical images of each modality.

[0049] Specifically, each modality of 3D medical imaging has preset weights for each type of tissue (i.e., modality weights). For example, each modality of medical imaging includes MRI, DSA, and ultrasound imaging, and these modal medical imaging can segment "blood vessels", "tumors", and "backgrounds". The above modal medical imaging has preset weights for "blood vessels" tissues, and has other weights for "tumor" tissues. When fusion is performed, the preset weights of each type of tissue are combined with each modality of medical imaging to make the fusion result more accurate.

[0050] The above weights are flexibly set according to the characteristics of medical images. For example, DSA has a greater weight for "vascular" tissue than MRI and ultrasound imaging, and MRI has a greater weight for "tumor" tissue than DSA and ultrasound imaging. By presetting the weights, the features can be retained to a greater extent. The above preset weights can also be compatible with the lack of modality. For example, DSA examinations take a long time, and only MRI and ultrasound images are collected before treatment. In this case, only the segmentation results of MRI and ultrasound images can be fused.

[0051] This embodiment trains a deep learning model for each modality of medical images. Each model has a stronger ability to segment medical images of the corresponding modality and a higher segmentation accuracy. Different types of medical images design their own preset weights (modality weights) for each type of tissue. In the fusion process, the segmentation results are fused according to the modality weights, which reflects the characteristics of the tissue structure to a greater extent and improves the multi-modal segmentation accuracy. The above fusion method is also compatible with the lack of modality, expanding the scope of application.

[0052] Based on the previous embodiment, in one embodiment, the three-dimensional medical images of multiple modalities in step S1 include cross-modal images generated using a large model. Specifically, the large model is a cross-modal image generation model, which has the ability to "generate another modality medical image based on an existing modality medical image" after pre-training. For example, in one scene, a patient's T1 image is collected, and the T1 image is input into a pre-trained cross-modal image generation model, and the model generates a T2 image of the patient. Thereafter, the "T1 image, T2 image" can be used for subsequent multi-modal tissue segmentation and fusion. For another example, in another scene, a patient's MRI image and CT image are collected, and a CTA image is missing. The patient's MRI image and CT image are input into a trained cross-modal image generation model, and the model can generate the patient's CTA image. Thereafter, the "MRI image, CT image, CTA image" can be used for subsequent multi-modal tissue segmentation and fusion, and the tissue structure characteristics are more accurately and comprehensively presented in the fused image, providing information reference for doctors.

[0053] Preferably, the cross-modal image generation model converts between different modalities acquired by the same type of equipment to enhance the quality of the generated image. For example, T1 images and T2 images are both images acquired by magnetic resonance equipment, and the cross-modal image generation model converts the modality between T1 and T2. For another example, EBT and PET-CT are both images acquired by CT equipment, and the cross-modal image generation model converts the modality between EBT and PET-CT.

[0054] Based on any embodiment, in one embodiment, S3 includes: performing weighted fusion on the three-dimensional segmentation results of each modality according to tissue category.

[0055] Specifically, this embodiment provides a fusion method to fuse the segmentation results of each modality according to the tissue category. For example, the patient's three-dimensional medical images include CT, MRI, DSA, PET, and ultrasound, and each of the above images can segment "tumors" respectively, among which MRI, DSA, and ultrasound can also segment "blood vessels". During fusion, the "tumor" segmentation results of each modality are weighted and fused according to the preset weights of CT, MRI, DSA, PET, and ultrasound images for "tumors", and the "blood vessels" segmentation results of each modality are weighted and fused according to the preset weights of MRI, DSA, and ultrasound images for "blood vessels".

[0056] It is understandable that the weight combination of each modality of "tumor" is different from the weight combination of each modality of "blood vessel", and the weighted fusion process of the two is not the same. The impact factors of different modal medical images on different tissue types are reasonably adjusted through preset weights, making the fusion results more reasonable, accurate and comprehensive.

[0057] This embodiment provides a fusion method, which fuses the segmentation results of each modality for each type of tissue by modality weighting (that is, the fusion results can be managed separately by tissue type). Doctors can switch to view different types of tissue structures, which is convenient for viewing the structure and morphology of specific tissues (such as observing the morphology and size of lesions).

[0058] Based on any embodiment, in one embodiment, S3 further includes:

[0059] For each type of tissue, determine the medical image set that can be used to segment the tissue, and obtain the preset weight of each modality of medical image for the tissue;

[0060] According to the preset weights of each modality of medical images in the medical image set for this type of tissue, and based on the registration relationship, weighted fusion is performed on the 3D segmentation results of each modality of this type of tissue to obtain the weighted probability that the fused voxel belongs to this type of tissue;

[0061] The weighted probability that the fused voxel belongs to the type of tissue is compared with a preset probability threshold of the tissue to determine whether the fused voxel belongs to the type of tissue.

[0062] Specifically, in practical applications, only partial modality medical images of the patient may be acquired. The method of the present invention is compatible with the situation of lack of modality, and can still segment and fuse partial modality medical images.

[0063] For example, Table 1 below shows an example of preset weights of various medical images for various tissues:

[0064]

[0065] Table 1

[0066] The patient was imaged with CT, MRI, DSA, and ultrasound, but lacked PET modality. The fusion process is as follows:

[0067] For the “vascular” tissue, the medical image set that can currently segment the “vascular” is determined to be MRI, DSA, and ultrasound from the above-mentioned multiple modalities of medical images, and the corresponding preset weights are 3, 6, and 1, respectively.

[0068] According to the above weights, the "blood vessels" segmented from MRI, DSA, and ultrasound images are weighted and fused. For example, according to the registration relationship, the segmentation result of MRI at a certain voxel is determined to be "blood vessel", the segmentation result of DSA is "blood vessel", and the segmentation result of ultrasound is "non-blood vessel", then the weighted probability of this fused voxel is: (1×3+1×6+0×1) / 10=0.9, and it is determined that 0.9 is greater than the preset probability threshold of "blood vessel" 0.5, and the fused voxel belongs to "blood vessel". Referring to this process, it can be determined in turn whether other voxels belong to "blood vessel", thereby obtaining the "blood vessel" fusion result.

[0069] For the "tumor" tissue, the modalities that can currently segment the "tumor" are determined to be CT, MRI, DSA, and ultrasound, and the corresponding preset weights are 1, 3, 4, and 1, respectively.

[0070] According to the above weights, the "tumors" segmented from CT, MRI, DSA, and ultrasound images are fused. For example, according to the registration relationship, the segmentation result of CT at a certain voxel is determined to be "tumor", the segmentation result of MRI is "tumor", the segmentation result of DSA is "tumor", and the segmentation result of ultrasound is "non-tumor". Then the weighted probability of this fused voxel is: (1×1+0×3+1×4+0×1) / 9=0.55. It is determined that 0.66 is greater than the preset probability threshold of "tumor" 0.6, and the fused voxel belongs to "non-tumor". Referring to this process, it can be determined in turn whether other voxels belong to "tumor", thereby obtaining the "tumor" fusion result. Similarly, other tissues such as "skull" can also be weighted fused.

[0071] This embodiment realizes the fusion of various types of tissues, and users can easily view different types of tissue structures. In the fusion process, this embodiment fuses the segmentation results of each modality for each type of tissue by modality weighting, and can manage the fusion results separately by tissue type. Doctors can easily switch to view different types of tissue structures, such as observing the morphology and size of specific lesion tissues.

[0072] Based on any embodiment, in one embodiment, S3 includes:

[0073] The fusion is performed voxel by voxel according to the registration relationship, and the weighted probability of the fused voxel belonging to each tissue type is determined by modality weighting;

[0074] The tissue type corresponding to the maximum weighted probability is determined as the fusion label of the voxel.

[0075] Specifically, this embodiment provides another weighted fusion method to achieve voxel-level fusion and observe various types of tissues in three-dimensional images. During the fusion process, the weighted probability of the fused voxel belonging to each type of tissue is calculated in combination with the aforementioned preset weights. The greater the weighted probability, the higher the possibility that the fused voxel belongs to the corresponding tissue type. The tissue type corresponding to the maximum weighted probability is determined as the fusion label of the voxel. Furthermore, different types of tissues are rendered in different colors in the fused image for easy viewing.

[0076] This embodiment provides another fusion method, which reasonably adjusts the influence factors of different modal medical images on different tissue types through preset weights, so that the calculation of weighted probability is more reasonable and accurate; by comparing the weighted probabilities of various tissue types, the labels after voxel fusion are determined, making the fusion result more reasonable, especially improving the fusion accuracy at the junction of different types of tissues; the tissue structure that can be displayed in the fusion image makes it convenient for doctors to view the spatial position relationship of different tissues, and facilitates the discussion and planning of surgical plans.

[0077] Based on any embodiment, in one embodiment, determining the weighted probability that the fused voxel belongs to each tissue type by modality weighting includes:

[0078] Determine the currently available medical image set and obtain the preset weights of each modality of medical image for each type of tissue;

[0079] According to the modal weights of each modality of medical images in the medical image set for this type of tissue, and based on the registration relationship, weighted fusion is performed on the segmentation results of each modality of the first tissue to obtain a weighted probability that the fused voxel belongs to the first tissue; wherein the first tissue is one of multiple tissue types.

[0080] Specifically, in practical applications, only partial modality medical images of the patient may be acquired. The method of the present invention is compatible with the situation of lack of modality, and can still segment and fuse partial modality medical images.

[0081] For example, Table 2 below shows another example of preset weights of various medical images for various tissues:

[0082]

[0083] Table 2

[0084] The patient acquired CT, DSA, PET, and ultrasound images, but lacked MRI modality. The fusion process is as follows:

[0085] The preset weights of CT, DSA, PET, and ultrasound imaging modalities for “skull,” “blood vessel,” “tumor,” and “brain functional area” are obtained from Table 2 , respectively.

[0086]

[0087]

[0088] Table 3

[0089] Table 3 illustrates the segmentation results of a certain voxel under each modality image. The segmentation results are weighted according to the preset weights obtained to obtain the weighted probability that the voxel belongs to each type of tissue:

[0090] Weighted probability of belonging to the skull: (0×8+0×2) / 10=0;

[0091] Weighted probability of belonging to a blood vessel: (1×5+1×1) / 6=1;

[0092] Weighted probability of belonging to a tumor: (0×1+0×1+0×4+0×0.5) / 6.5=0;

[0093] Weighted probability of belonging to brain functional area: (0×2) / 2=0;

[0094] The weighted probability of belonging to the background: (1×1+0×1+1×1+0×1) / 4=0.5.

[0095] By comparison, it is determined that the weighted probability that the voxel belongs to the blood vessel is the largest, and the fusion result of the voxel is determined as "blood vessel". Referring to this process, the fusion results of other voxels are determined respectively, and finally the fusion result of each modality segmentation result is obtained.

[0096] In this embodiment, different preset weights are assigned to each modality of medical images for different tissue types. The preset weights are used to reasonably adjust the impact factors of different modalities of medical images on different tissue types, so that the calculation of weighted probability is more reasonable and the fusion result is more accurate, especially improving the fusion accuracy at the tissue junction.

[0097] Based on any embodiment, in one embodiment, S3 further includes: according to the registration relationship, fusing and transforming the three-dimensional segmentation results of each modality into the same coordinate system, so as to view the segmentation results of each modality.

[0098] This embodiment provides another fusion method based on the above fusion method. Specifically, the segmentation results of each modality are converted to the same coordinate system according to the registration relationship, and the doctor can switch to view the segmentation results of each type of tissue under each modality. It can be understood that different modalities of medical images can segment different tissue types, and the segmentation accuracy of different tissues is also different. The initial segmentation results of each modality provide more basic information, which is convenient for doctors to further analyze the tissue structure.

[0099] Based on any embodiment, in one embodiment, S2 includes:

[0100] The first modality medical image and the second modality medical image are unified in size; based on maximum mutual information, the first modality medical image and the second modality medical image are registered to obtain a registration relationship between the two.

[0101] For example, the same size process can unify the medical images of each modality to the size of a certain modality. For another example, unify the medical images of each modality to the same standard size. The same modality medical image size can improve the registration accuracy and reduce the registration calculation amount. The registration process can be based on the maximum mutual information. The mutual information can be used to describe the similarity of two images, and the registration parameter with the maximum mutual information is the final registration relationship. The mutual information can be calculated based on the grayscale histogram or the probability calculation.

[0102] This embodiment can guide the fusion process and achieve high-precision fusion by unifying the size of medical images of each modality and obtaining a registration relationship based on maximum mutual information registration.

[0103] Based on any one of the embodiments, in one embodiment, S2 further includes: using the segmentation structure of the first modality medical image and the segmentation result of the second modality medical image to guide the registration process of the first modality medical image and the second modality medical image.

[0104] Specifically, in this embodiment, step S1 is performed before step S2. Since the original images of different modalities are quite different and the registration accuracy is low, after the image is segmented in step S1, the segmentation result is used to guide the registration process of step S2, for example, edge features are extracted from the segmentation result and added to the original image to guide the registration process, so that the registration accuracy is higher, and the registered image is more consistent with the actual condition of the patient, which is conducive to improving the fusion accuracy.

[0105] The present invention also provides a surgical navigation system. The surgical navigation system described below and the multimodal tissue segmentation method described above can be referenced to each other.

[0106] Figure 2 is a structural schematic diagram of a surgical navigation system provided by the present invention, such as Figure 2 As shown, the surgical navigation system includes: a host 21 and a display device 22.

[0107] The host 100 stores executable instructions, and when the executable instructions are executed by the host 100, any of the multimodal tissue segmentation methods described above is implemented.

[0108] The display device 22 is used to display medical images, interactive interfaces, fusion results, etc., for doctors to view.

[0109] This embodiment can provide information support for preoperative surgical plan planning through multimodal tissue segmentation. During the surgical navigation process, it can further present the patient's three-dimensional model / surgical plan / surgical instruments and other contents on the display device 22 to guide the surgical process.

[0110] The present invention also provides a computer program product, which includes computer executable instructions, and the instructions are used to implement any of the multimodal tissue segmentation methods described above when executed.

[0111] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which implements any of the aforementioned multimodal tissue segmentation methods when executed by a processor.

[0112] The system embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, i.e., they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative effort.

[0113] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods of each embodiment or some parts of the embodiment.

[0114] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multimodal tissue segmentation method, characterized in that: include: Input three-dimensional medical images of various modalities into corresponding deep learning network models for segmentation, and obtain corresponding three-dimensional segmentation results; Register the three-dimensional medical images of each modality to obtain the registration relationship; According to the registration relationship, the three-dimensional segmentation results of each modality are weighted and fused in combination with the preset weights of each type of tissue in the three-dimensional medical images of each modality.

2. The multimodal tissue segmentation method according to claim 1, characterized in that: The three-dimensional segmentation results of each modality are weighted and fused according to the registration relationship and the preset weights of each type of tissue in the three-dimensional medical images of each modality, including: weighted fusion of the three-dimensional segmentation results of each modality according to the tissue category.

3. The multimodal tissue segmentation method according to claim 2, characterized in that: The weighted fusion of the three-dimensional segmentation results of each modality based on the registration relationship and the preset weights of each type of tissue in the three-dimensional medical images of each modality further includes: For each type of tissue, determine the medical image set that can be used to segment the tissue, and obtain the preset weight of each modality of medical image for the tissue; According to the preset weights of each modality of medical images in the medical image set for the type of tissue, and based on the registration relationship, weighted fusion is performed on the three-dimensional segmentation results of each modality of the type of tissue to obtain a weighted probability that the fused voxel belongs to the type of tissue; The weighted probability that the fused voxel belongs to the type of tissue is compared with a preset probability threshold of the tissue to determine whether the fused voxel belongs to the type of tissue.

4. The multimodal tissue segmentation method according to claim 1, characterized in that: The weighted fusion of the three-dimensional segmentation results of each modality according to the registration relationship and the preset weight of each type of tissue in the three-dimensional medical images of each modality includes: fusing each voxel according to the registration relationship, and determining the weighted probability that the fused voxel belongs to each tissue type by means of modality weighting; The tissue type corresponding to the maximum weighted probability is determined as the fusion label of the voxel.

5. The multimodal tissue segmentation method according to claim 4, characterized in that: The method of determining the weighted probability that the fused voxel belongs to each tissue type by modality weighting includes: Determine the currently available medical image set and obtain the preset weights of each modality of medical image for each type of tissue; According to the modal weights of each modality of medical images in the medical image set for this type of tissue, and based on the registration relationship, weighted fusion is performed on the segmentation results of each modality of the first tissue to obtain a weighted probability that the fused voxel belongs to the first tissue; wherein the first tissue is one of multiple tissue types.

6. The multimodal tissue segmentation method according to any one of claims 1 to 5, characterized in that: The weighted fusion of the three-dimensional segmentation results of each modality based on the registration relationship and the preset weights of each type of tissue in the three-dimensional medical images of each modality also includes: according to the registration relationship, the fusion and conversion of the three-dimensional segmentation results of each modality into the same coordinate system so as to view the segmentation results of each modality.

7. The multimodal tissue segmentation method according to claim 1, characterized in that: The registering of the three-dimensional medical images of each modality to obtain a registration relationship includes: unifying the size of the first modality medical image and the second modality medical image; Based on the maximum mutual information, the first modality medical image and the second modality medical image are registered to obtain a registration relationship between the two.

8. The multimodal tissue segmentation method according to claim 7, characterized in that: The registering of the three-dimensional medical images of each modality to obtain a registration relationship also includes: using the segmentation structure of the first modality medical image and the segmentation result of the second modality medical image to guide the registration process of the first modality medical image and the second modality medical image.

9. A computer program product, comprising computer executable instructions, characterized in that: When executed, the instructions are used to implement the multimodal tissue segmentation method as described in any one of claims 1-8.

10. A surgical navigation system, comprising a host and a display device, characterized in that: The host stores executable instructions, and when the executable instructions are executed by the host, the multimodal tissue segmentation method according to any one of claims 1 to 8 is implemented.

Images