Multi-task bone joint ct / mri fusion tissue precise segmentation modeling method and device

Abstract

The multi-task bone joint CT / MRI fusion tissue precise segmentation modeling method and device can realize image noise point interference removal under different modalities, get rid of manual labeling tissue segmentation operation, and improve bone joint bone tissue and soft tissue segmentation and modeling accuracy. The method comprises the following steps: (1) for clinical medical image CT / MRI, image enhancement is carried out through cascade supervision denoising and weak supervision; (2) the enhanced clinical image is fed back to the edge attention supervision for hard tissue segmentation; (3) soft tissue segmentation is realized through double priori supervision, and the hard tissue segmentation result generated by the former is used as priori guidance.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of medical image analysis and machine learning, and in particular to a multi-task bone joint CT / MRI fusion tissue accurate segmentation modeling method and a multi-task bone joint CT / MRI fusion tissue accurate segmentation modeling device. BACKGROUND

[0002] Joint replacement surgery, as an important method for treating rheumatoid arthritis and other bone and joint diseases, can effectively solve the problems of joint pain, bone deformity, limited range of motion, etc. Traditional artificial joint replacement surgery relies on the surgical tools, surgical experience and communication effect of the operator, and is prone to problems such as inaccurate placement of joint prostheses, leading to a series of postoperative sequelae such as prosthesis displacement and wear.

[0003] Based on the above problems, robot-assisted joint replacement surgery has emerged. Compared with traditional manual operation, robot-assisted joint replacement surgery can achieve more accurate prosthesis implantation, better protection of soft tissue structures around the joint, shorter learning curve for the operator, and higher patient satisfaction after surgery. In current clinical operations, preoperative planning is based on medical workers' segmentation of bone and joint tissues and three-dimensional modeling of CT / MRI images through manual interaction.

[0004] In recent years, with the rapid development of machine learning technology, especially deep learning technology, deep learning has become an important tool in the field of image analysis. Compared with traditional manual medical image analysis, deep learning-based methods can avoid the subjectivity and limitations of image analysis personnel, speed up the analysis, and improve the analysis accuracy. Currently, deep learning methods have been widely used in medical image detection, classification, segmentation, reconstruction, synthesis and other tasks.

[0005] The noise commonly existing in clinical medical images, such as CT low-dose perfusion noise, MR Rayleigh noise and Gaussian noise, can interfere with the topological structure and edge representation of various tissues, causing the deep neural network model to extract incorrect features; and artificial annotation differences can affect the training of the model, which is not conducive to parameter convergence, thereby reducing the segmentation accuracy and robustness of the model. SUMMARY

[0006] To overcome the defects of the prior art, the technical problem to be solved by the present application is to provide a multi-task bone joint CT / MRI fusion tissue accurate segmentation modeling method, which can remove noise interference, eliminate manual operation, and improve the accuracy of bone and joint bone tissue and soft tissue segmentation and modeling.

[0007] The technical solution of the present application is: this multi-task bone joint CT / MRI fusion tissue accurate segmentation modeling method comprises the following steps:

[0008] (1) For clinical medical image CT / MRI, image enhancement is performed through cascading supervised denoising and weak supervision;

[0009] (2) The enhanced clinical image is fed back to the edge attention supervision for hard tissue segmentation;

[0010] (3) Soft tissue segmentation is performed through double prior supervision, and the latter needs the hard tissue segmentation result generated by the former as prior guidance.

[0011] The application uses a deep learning method, and proposes a complete set of solutions for preoperative bone joint tissue segmentation and modeling for joint replacement surgery. Firstly, data augmentation is performed for the problems of large noise in clinical medical images and large differences in manual annotation; then the features of various tissues are extracted through a deep neural network, and the bone tissue segmentation result and the soft tissue anatomy feature are used as prior information to realize accurate and efficient segmentation of bone tissue and complex soft tissue; finally, a three-dimensional model is generated based on the bone joint tissue segmentation model. Therefore, noise interference can be removed, manual operation can be eliminated, and the accuracy of bone joint bone tissue and soft tissue segmentation and modeling can be improved.

[0012] The application provides a multi-task bone joint CT / MRI fusion tissue accurate segmentation modeling network module, which comprises:

[0013] An image enhancement module configured to enhance the image through cascading supervised denoising and weak supervision for clinical medical image CT / MRI;

[0014] A hard tissue segmentation module for feeding back the enhanced clinical image to the edge attention supervision for hard tissue segmentation;

[0015] A soft tissue segmentation module for performing soft tissue segmentation through double prior supervision, and the latter needs the hard tissue segmentation result generated by the former as prior guidance. BRIEF DESCRIPTION OF DRAWINGS

[0016] Figure 1 is a flowchart of the multi-task bone joint CT / MRI fusion tissue accurate segmentation modeling method according to the application.

[0017] Figure 2 is a flowchart of noise removal based on a cascading hierarchical supervision mechanism according to the application.

[0018] Figure 3 is a flowchart of bone joint segmentation label data enhancement based on a weak supervision mechanism according to the application.

[0019] Figure 4is a flowchart of the edge attention supervision guided bony key soft tissue segmentation based on the application.

[0020] Figure 5 is a flowchart of the bony structure edge feature extraction based on the double attention encoding according to the application.

[0021] Figure 6 is a flowchart of the key soft tissue segmentation guided by fusing bony tissue segmentation and soft tissue anatomy prior knowledge according to the application. DETAILED DESCRIPTION

[0022] As shown in Figure 1 , the multi-task bone joint CT / MRI fusion tissue precise segmentation modeling method includes the following steps:

[0023] (1) For clinical medical image CT / MRI, image enhancement is carried out through cascaded supervision denoising and weak supervision;

[0024] (2) The enhanced clinical image is fed back to the edge attention supervision for hard tissue segmentation, and the soft tissue segmentation is carried out through double prior supervision, which needs the hard tissue segmentation result generated by the former as prior guidance;

[0025] (3) The hard tissue segmentation result and the soft tissue segmentation result are fused and input to carry out point cloud local feature description key anatomy and force line calculation to generate lower limb force line positioning result.

[0026] The application uses a deep learning method, faces joint replacement surgery, and proposes a complete set of multi-task bone joint CT / MRI tissue precise segmentation solutions to get rid of manual operation for preoperative bone joint tissue segmentation and modeling. Firstly, for the problem of large noise and large manual annotation difference of clinical medical image, data enhancement is carried out; then the features of various tissues are extracted through a deep neural network, and the bony tissue segmentation result and the soft tissue anatomy feature are used as prior information to realize precise and efficient segmentation of bony tissue and complex soft tissue; finally, based on the bone joint tissue segmentation model, a three-dimensional model is generated. Therefore, it can realize noise point interference removal, get rid of manual operation, and improve the accuracy of bone joint bone tissue and soft tissue segmentation and modeling.

[0027] Preferably, the step (1) comprises the following sub-steps:

[0028] (1.1) image enhancement based on a multi-modal image cascaded multi-element supervision mechanism;

[0029] (1.2) bone joint segmentation label data enhancement based on a weak supervision guidance mechanism.

[0030] Preferably, in step (1.1), the input preoperative MR image is first fed back to the cascaded hierarchical supervision module to learn to extract noise features of different types and scales in the image, then the noise features are fused, the mixed image is obtained by pixel-by-pixel subtraction of the original MR image and the noise feature map, and finally the denoised MR image is obtained by adaptive convolution filtering and down-sampling filtering of the mixed image.

[0031] Preferably, in step (1.2), the input image is represented by a corresponding attention map generated by weakly supervised attention learning to represent the significant feature part of the image, and then the input image is enhanced by randomly selecting an attention map using attention cropping and attention deletion, the input image and the enhanced image are used as iterative training data, the class probability of the target and the attention map are output by weakly supervised learning, the target position is located and refined, and finally the data of the previous two stages are fused in the channel to output the learned bone joint data enhanced image.

[0032] Preferably, in step (2), edge detection segmentation and semantic segmentation are integrated into a network, a semantic segmentation network with an edge attention module is designed, and the segmentation process is guided based on edge attention; the network guides the encoder to learn different scale bone tissue feature representations based on a dual attention module, and guides the decoder to aggregate multi-scale feature representations using a weighted aggregation module, so as to learn high-level semantic features while preserving low-level local edge features and improve the segmentation accuracy of the module for different types of bone tissue; wherein the dual attention module integrates spatial attention mechanism and channel attention mechanism to enhance the feature representation of bone structure and obtain a saliency feature map.

[0033] Preferably, step (3) comprises the following sub-steps:

[0034] (3.1) extracting and fusing based on soft tissue anatomical information and bone tissue segmentation information;

[0035] (3.2) soft tissue segmentation guided by soft tissue anatomical prior and bone tissue segmentation prior.

[0036] Preferably, in the step (3.1), a multi-channel feature fusion framework based on multi-scale feature images is established using a deep convolutional neural network model with reverse connection combined with multi-scale modal feature fusion to extract key soft tissue anatomical feature information and bone tissue segmentation feature information; multi-scale convolution operation and pooling operation are performed in parallel for the input image, and all calculation results are spliced into a deep feature map; during the forward convolution of the image, the previous convolution calculation result is simultaneously input as part of the subsequent convolution for feature fusion; the input image is converted into a high-dimensional feature image by using the convolution layer and the pooling layer, and the high-dimensional features are extracted at the same time, the converted high-dimensional feature image is input into the previous convolution layer as input by using the reverse connection, and the feature image of the current layer is reduced to the dimension size of the previous layer by using the deconvolution and the up-pooling operation, and the feature fusion is performed by cascading the features of the original input; based on the above multi-scale feature extraction and reverse connection operation, the deep soft tissue anatomical feature information and the bone tissue segmentation feature information are extracted and fused.

[0037] Preferably, in the step (3.2), the bone joint MR image and the bone tissue segmentation result are input into two different prior feature encoders as inputs, the prior segmentation feature information of the bone tissue and the soft tissue anatomical prior segmentation information in different dimensions are input into the local supervision subnetwork as network guide by the way of multi-element feature cascade, the multi-element feature fusion is performed between different subnetworks, and finally the key soft tissue structure segmentation image is output.

[0038] Those skilled in the art can understand that all or part of the steps in the above-mentioned embodiment methods can be completed by programs instructing related hardware, and the programs can be stored in a computer-readable storage medium. When the program is executed, each step of the above-mentioned embodiment method is included, and the storage medium can be ROM / RAM, a magnetic disc, an optical disc, a memory card, etc. Therefore, corresponding to the method of the present application, the present application also simultaneously includes a multi-task bone joint CT / MRI fusion tissue precise segmentation modeling device. The device comprises:

[0039] An image enhancement module configured to perform image enhancement by cascading supervised denoising and weak supervision for clinical medical image CT / MRI;

[0040] A hard tissue segmentation module that feeds back the enhanced clinical image to edge attention supervision for hard tissue segmentation;

[0041] A soft tissue segmentation module that performs soft tissue segmentation by double prior supervision, which needs the hard tissue segmentation result generated by the former as prior guidance.

[0042] Preferably, the hard tissue segmentation module performs:

[0043] (1.1) Image enhancement based on multi-modal image cascade multi-element supervision mechanism;

[0044] (1.2) Bone joint segmentation label data enhancement based on weakly supervised guidance mechanism;

[0045] The hard tissue segmentation module performs: integrating edge detection segmentation and semantic segmentation into one network, designing a semantic segmentation network with an edge attention module, and guiding the segmentation process based on edge attention; the network guides the encoder to learn bone tissue feature representations of different scales based on a dual attention module, and guides the decoder to aggregate multi-scale feature representations using a weighted aggregation module, while learning high-level semantic features, the low-level local edge features are preserved, and the segmentation accuracy of the module for different types of bone tissue is improved; wherein the dual attention module fuses spatial attention mechanism and channel attention mechanism, enhances the feature representation of the bone structure, and obtains a saliency feature map;

[0046] The soft tissue segmentation module performs:

[0047] (3.1) Extraction and fusion based on soft tissue anatomical information and bone tissue segmentation information;

[0048] (3.2) Soft tissue segmentation guided by soft tissue anatomical prior and bone tissue segmentation prior.

[0049] The above is only a preferred embodiment of the present application, and does not limit the present application in any form. Any simple modification, equivalent change and modification made according to the technical essence of the present application to the above embodiment are still within the protection scope of the technical solution of the present application.

Claims

1. A multi-task bone joint CT / MRI fusion tissue precise segmentation modeling method, characterized in that: It comprises the following steps: (1) For clinical medical CT / MRI images, noise interference is eliminated by cascading supervised denoising and weakly supervised image enhancement; (2) The enhanced clinical images are fed back to the semantic segmentation network of the fusion edge attention module for hard tissue segmentation; (3) Soft tissue segmentation is performed by double prior supervision, which requires the hard tissue segmentation results generated by the former as prior guidance; Step (1) comprises the following steps: (1.1) image enhancement based on multi-modal image cascading multi-element supervision mechanism; (1.2) bone joint segmentation label data enhancement based on weakly supervised guidance mechanism; In step (1.2), the input image generates a corresponding attention map to represent the significant feature part of the image through weakly supervised attention learning, and then uses attention cropping and attention deletion to enhance the input image. The input image and the enhanced image are used as iterative training data, and the target class probability and attention map are output through weakly supervised learning. Then the target position is located and refined, and the data of the previous two stages are fused in the channel to output the learned bone joint data enhancement image. In step (2), edge detection segmentation and semantic segmentation are integrated into a network, and a semantic segmentation network with a fusion edge attention module is designed to guide the segmentation process based on edge attention. The network uses a double attention module to guide the encoder to learn bone tissue feature representations at different scales, and uses a weighted aggregation module to guide the decoder to aggregate multi-scale feature representations. While learning high-level semantic features, it retains low-level edge features to improve the segmentation accuracy of the module for different types of bone tissue. The double attention module combines spatial attention mechanism and channel attention mechanism to enhance the feature representation of bone structure and obtain a significant feature map.

2. The multi-task bone joint CT / MRI fused tissue precise segmentation modeling method according to claim 1, characterized in that: In step (1.1), the input preoperative MR image is first fed back to the cascaded hierarchical supervision module to learn and extract different types and scales of noise features in the image. Then, the noise features are fused, and the mixed image is obtained by pixel-by-pixel subtraction of the original MR image and the noise feature map. Finally, the mixed image is down-sampled and filtered by adaptive convolution filtering to obtain the denoised MR image.

3. The multi-task bone joint CT / MRI fused tissue precise segmentation modeling method according to claim 2, characterized in that: Step (3) comprises the following steps: (3.1) extraction and fusion of soft tissue anatomical information and bone tissue segmentation information; (3.2) soft tissue segmentation based on soft tissue anatomical prior and bone tissue segmentation prior guidance; In step (3.2), the bone joint MR image and the bone tissue segmentation result are input into two different prior feature encoders as inputs. The segmentation prior information of bone tissue at different dimensions and the anatomical prior information of soft tissue are input into the local supervision subnetwork as network guidance through multi-element feature cascading. The multi-element features are fused between different subnetworks, and finally the key soft tissue structure segmentation image is output.

4. The multi-task bone joint CT / MRI fused tissue precise segmentation modeling method according to claim 3, characterized in that: In the step (3.1), a multi-channel feature fusion framework based on multi-scale feature images is established by using a deep convolutional neural network model with reverse connection and combining multi-scale modal feature fusion to extract key soft tissue anatomical feature information and bone tissue segmentation feature information; multi-scale convolution operation and pooling operation are performed in parallel for the input image, and all calculation results are spliced into a deep feature map; in the process of forward convolution of the image, the front convolution calculation result is simultaneously input as part of the back convolution to perform feature fusion; the input image is converted into a high-dimensional feature image by using the convolution layer and the pooling layer, and the high-dimensional features are extracted; the converted high-dimensional feature image is input into the previous convolution layer as input by using the reverse connection; the feature image of the current layer is reduced to the dimension size of the previous layer by using the deconvolution and the upper pooling operation, and feature fusion is performed on the feature channels with the original input; based on the above multi-scale feature extraction and reverse connection operation, deep soft tissue anatomical feature information and bone tissue segmentation feature information are extracted and fused.

5. The multi-task bone joint CT / MRI fused tissue precise segmentation modeling method device according to claim 1, characterized in that: It comprises: An image enhancement module configured to perform image enhancement on clinical medical images CT / MRI through cascaded supervised denoising and weak supervision; A hard tissue segmentation module configured to feed the enhanced clinical images to edge attention supervision for hard tissue segmentation; A soft tissue segmentation module configured to perform soft tissue segmentation through double priori supervision, which needs the hard tissue segmentation result generated by the former as priori guidance.

6. The multi-task bone joint CT / MRI fused tissue precise segmentation modeling method device according to claim 5, characterized in that: The image enhancement module performs: (1) image enhancement based on a multi-modal image cascaded multi-element supervision mechanism; (2) bone joint segmentation label data enhancement based on a weak supervision guidance mechanism; The hard tissue segmentation module performs: integrating edge detection segmentation and semantic segmentation into one network, designing a semantic segmentation network with edge attention mechanism, and guiding the segmentation process based on edge attention; the network learns different scale bone tissue feature representations based on double attention guided encoder, and uses a weighted aggregation module to guide the decoder to aggregate multi-scale feature representations, while learning high-level semantic features and preserving low-level local edge features, thereby improving the segmentation accuracy of different types of bone tissue by the network; Wherein the double attention module fuses spatial attention mechanism and channel attention mechanism to enhance the feature representation of bone structure, and finally obtains a significant feature map; The soft tissue segmentation module performs: (3.1) extraction and fusion based on soft tissue anatomical information and bone tissue segmentation information; (3.2) soft tissue segmentation based on soft tissue anatomical priori and bone tissue segmentation priori guidance.

Images