A multi-modal fusion medical image segmentation method

Abstract

The application relates to the field of medical image processing, in particular to a multi-modal fusion medical image segmentation method. The method comprises the following steps: determining a multi-modal original image; performing classification processing to obtain a classified image set; performing preprocessing on the classified image set to obtain a preprocessed data set; training a CNN network structure through the preprocessed data set to obtain a part recognition combined model; training a U-Net network structure through the preprocessed data set to obtain a segmentation combined model; obtaining a multi-modal to-be-segmented picture, pre-processing the to-be-segmented picture to obtain a to-be-input picture corresponding to the to-be-segmented picture, and sequentially inputting the recognition combined model and the segmentation combined model to obtain a segmented picture. The part classification combined model is arranged to firstly classify the multi-modal original picture to obtain a part name, and then the segmentation combined model is used to perform weight segmentation processing on the part according to the part name, so that a segmentation result is finally obtained.

Description

Technical Field

[0001] This application relates to the field of medical image processing, and in particular to a multimodal fusion method for medical image segmentation. Background Technology

[0002] Automatic segmentation of medical images is a crucial step in image processing and analysis, playing a vital role in clinical diagnosis and treatment. Multimodal magnetic resonance imaging (MRI) provides more information than single-modal data, and different modal MRIs are often combined in neuroimaging research to overcome the limitations of single-modal imaging techniques. Therefore, understanding how to fuse multimodal MRI information to improve automatic segmentation performance is significant for clinical diagnosis and treatment.

[0003] For medical images, using only a single modality may lead to misjudgment or missed detection. Making full use of the complementary information between different modalities of medical images can improve the detection rate and segmentation accuracy.

[0004] However, existing multimodal medical image segmentation methods are widely used, but they cannot perform feature identification and segmentation for diseases in different parts of the body, resulting in limited segmentation accuracy. Summary of the Invention

[0005] This application provides a multimodal fusion medical image segmentation method that can solve the problem of existing segmentation models identifying and segmenting different body parts.

[0006] The technical solution of this application is a multimodal fusion medical image segmentation method, including:

[0007] S1: Identify several multimodal raw images of several target diseases in several locations;

[0008] The original multimodal images are classified to obtain several classification image sets related to different body parts;

[0009] S2: Perform preprocessing on the target disease for the classification image set to obtain several preprocessed datasets, each containing several images to be input;

[0010] S3: Train the CNN network structure using a preprocessed dataset to obtain a combined model for part recognition that takes multimodal original images as input and part names as output.

[0011] S4: Train the U-Net network structure using a preprocessed dataset to obtain a segmentation combination model that takes the input image as the input item and the segmented image as the output item, and is connected to the part recognition combination model.

[0012] S5: Obtain several multimodal images to be segmented, preprocess the images to be segmented to obtain the corresponding input images, and sequentially input the corresponding input images to the part recognition combination model and the segmentation combination model to obtain the corresponding segmented images.

[0013] Optionally, step S2 includes:

[0014] S21: Normalize the multimodal original images in the classification image set to obtain a normalized image set including several normalized images.

[0015] S22: Perform preprocessing operations including cutting, rotation and elastic deformation on the normalized images in the normalized image set to obtain several preprocessed datasets, each containing several images to be input.

[0016] Optionally, step S3 includes:

[0017] S31: Determine the basic structure of the CNN and decompose the convolution kernels of the convolutional layers in the basic structure of the CNN to obtain the CNN network structure accordingly;

[0018] S32: Train several CNN network structures using several classification image sets respectively to obtain several basic models for partial recognition;

[0019] S33: Determine a dispersed association tree for the representative similarity of several parts;

[0020] S34: Based on the association tree, perform association connection processing on several basic models for part recognition to obtain the corresponding combination model for part recognition.

[0021] Optionally, step S31 includes:

[0022] S311: Determine the basic structure of the CNN;

[0023] S312: Perform order reduction decomposition on the convolutional kernels of the convolutional layers in the basic CNN structure to obtain several decomposed kernels;

[0024] S313: According to the wave-shaped amplification amplitude, several decomposition kernels are amplified to obtain several amplified kernels and the corresponding CNN network structure.

[0025] Optionally, step S33 includes:

[0026] S331: Determine the similarity between multimodal original images of several parts, and determine the similarity threshold;

[0027] S332: Determine the center point as the part whose similarity to the remaining parts exceeds the similarity threshold.

[0028] S333: Draw several scattering lines around the center point to represent similarity greater than the similarity threshold, with the other end of each scattering line correspondingly connected to one of the remaining parts;

[0029] S334: For the other ends of several scatter lines, draw scatter lines corresponding to the similarity greater than the similarity threshold.

[0030] S335: Repeat step S334 until a scattered association tree representing part similarity is formed.

[0031] Optionally, step S4 includes:

[0032] S41: Determine the initial training weights for different modal original images based on several target diseases for each part.

[0033] S42: Determine the basic structure of U-Net, including the encoding sub-model and the decoding sub-model;

[0034] Based on the initial training weights, a multi-attention mechanism is introduced into the decoding sub-model in the U-Net basic structure to obtain the U-Net network structure;

[0035] S43: Train the U-Net basic structure corresponding to different initial training weights by preprocessing the dataset to obtain the basic segmentation model corresponding to different parts;

[0036] S44: Based on the location, connect the basic model for location recognition and the basic model for segmentation to obtain a segmentation combination model that includes several basic models for location recognition.

[0037] Optionally, step S5 includes:

[0038] S51: Obtain several multimodal images to be segmented, preprocess the images to be segmented, and obtain the corresponding input images to be segmented;

[0039] S52: Input the image to be input into the part recognition combination model. Several basic part recognition models in the part recognition combination model will detect the image to be input in turn until the name of the part in the image to be input is determined.

[0040] S53: Input the image to be input to the segmentation base model connected to the part recognition base model that confirms the part name of the image to be input, and obtain the segmented image corresponding to the image to be segmented. Beneficial effects

[0041] This application sets up a part classification and combination model to first classify the multimodal original image to obtain part names, and then, based on the part names, performs weighted segmentation processing on the parts through a segmentation and combination model to finally obtain the segmentation result.

[0042] As can be seen from the above, this application can adapt to different parts and can perform weighted segmentation on multimodal original images of different parts, thus solving the problem of existing segmentation models recognizing and segmenting different parts. Attached Figure Description

[0043] To more clearly illustrate the technical solution of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0044] Figure 1 This is a flowchart illustrating the multimodal fusion-based medical image segmentation method in the embodiments of this application;

[0045] Figure 2 This is a flowchart illustrating step S2 in an embodiment of this application;

[0046] Figure 3 This is a flowchart illustrating step S3 in an embodiment of this application;

[0047] Figure 4 This is a flowchart illustrating step S4 in an embodiment of this application;

[0048] Figure 5 This is a flowchart illustrating step S5 in an embodiment of this application. Detailed Implementation

[0049] The embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described below do not represent all embodiments consistent with this application. They are merely examples of systems and methods consistent with some aspects of this application as detailed in the claims.

[0050] This application provides a multimodal fusion method for medical image segmentation, such as... Figure 1 As shown, Figure 1 This is a flowchart illustrating the multimodal fusion-based medical image segmentation method in this application, including:

[0051] S1: Identify several multimodal raw images of several target diseases in several locations.

[0052] The original multimodal images are classified to obtain several classification image sets related to different body parts.

[0053] Specifically, the multimodal raw images include at least two of the following: two-dimensional CT medical images, two-dimensional MRI medical images, two-dimensional ultrasound medical images, or two-dimensional PETCT medical images.

[0054] S2: Perform preprocessing on the target disease for the classification image set to obtain several preprocessed datasets, each containing several images to be input.

[0055] like Figure 2 As shown, Figure 2 This is a flowchart illustrating step S2 in an embodiment of this application. Step S2 includes:

[0056] S21: Normalize the multimodal original images in the classification image set to obtain a normalized image set including several normalized images.

[0057] S22: Perform preprocessing operations including cutting, rotation and elastic deformation on the normalized images in the normalized image set to obtain several preprocessed datasets, each containing several images to be input.

[0058] Specifically, classification processing is performed on CT training datasets or MRI training datasets.

[0059] S3: Train the CNN network structure using a preprocessed dataset to obtain a combined model for part recognition that takes multimodal original images as input and part names as output.

[0060] like Figure 3 As shown, Figure 3 This is a flowchart illustrating step S3 in an embodiment of this application. Step S3 includes:

[0061] S31: Determine the basic structure of the CNN and decompose the convolution kernels of the convolutional layers in the basic structure of the CNN to obtain the CNN network structure accordingly.

[0062] Step S31 includes:

[0063] S311: Determine the basic structure of the CNN.

[0064] S312: Perform order reduction decomposition on the convolutional kernels of the convolutional layers in the basic CNN structure to obtain several decomposed kernels.

[0065] S313: According to the wave-shaped amplification amplitude, several decomposition kernels are amplified to obtain several amplified kernels and the corresponding CNN network structure.

[0066] S32: Train several CNN network structures using several classification image sets to obtain several basic models for partial recognition.

[0067] Specifically, multiple downsampling operations are performed on the image data in the classification image set to identify the deep features of each image data through convolutional and pooling layers. Multiple upsampling operations are then performed on the downsampled image data to store the deep features back into the image data through upsampling and convolutional layers. The Adam classification optimizer is used for coarse image classification to obtain the classification result. Next, segmentation processing is performed on the image classification result: feature point data with pre-set confidence levels are selected from the deep features, bilinear interpolation is performed on the feature point data, and the category of the deep features is identified based on the calculated feature point data to obtain the final image segmentation result. A loss function is calculated based on the final image segmentation result and either a CT or MRI training dataset. The parameters of the CNN network structure are adjusted based on the loss function until the CNN network structure is successfully trained.

[0068] S33: Determine a dispersed association tree for the representative similarity of several parts.

[0069] Step S33 includes:

[0070] S331: Determine the similarity between multimodal original images of several parts, and determine the similarity threshold.

[0071] S332: Determine the center point as the part whose similarity to the remaining parts exceeds the similarity threshold.

[0072] S333: Draw several scattering lines around the center point to represent similarity greater than the similarity threshold, with the other end of each scattering line correspondingly connected to one of the remaining parts.

[0073] S334: For the portions at the other end of several scatter lines, draw scatter lines corresponding to similarity values ​​greater than a similarity threshold.

[0074] S335: Repeat step S334 until a scattered association tree representing part similarity is formed.

[0075] S34: Based on the association tree, perform association connection processing on several basic models for part recognition to obtain the corresponding combination model for part recognition.

[0076] S4: Train the U-Net network structure using a preprocessed dataset to obtain a segmentation combination model that takes the input image as input, the segmented image as output, and is connected to the part recognition combination model.

[0077] like Figure 4 As shown, Figure 4 This is a flowchart illustrating step S4 in an embodiment of this application. Step S4 includes:

[0078] S41: Determine the initial training weights for different modalities of the original images based on several target symptoms for each part.

[0079] S42: Determine the U-Net infrastructure, including the encoding sub-model and the decoding sub-model.

[0080] Based on the initial training weights, a multi-attention mechanism is introduced into the decoding sub-model in the U-Net basic structure to obtain the U-Net network structure.

[0081] S43: Train the U-Net basic structure corresponding to different initial training weights by preprocessing the dataset to obtain the segmentation basic model corresponding to different parts.

[0082] Specifically, the weight values ​​of different modalities in each part of the image are obtained by iteratively updating the loss function during training. As the network iterates and updates, the modal weight values ​​corresponding to different parts also need to be learned and updated during training.

[0083] S44: Based on the location, connect the basic model for location recognition and the basic model for segmentation to obtain a segmentation combination model that includes several basic models for location recognition.

[0084] S5: Obtain several multimodal images to be segmented, preprocess the images to be segmented to obtain the corresponding input images, and sequentially input the corresponding input images to the part recognition combination model and the segmentation combination model to obtain the corresponding segmented images.

[0085] like Figure 5 As shown, Figure 5 This is a flowchart illustrating step S5 in an embodiment of this application. Step S5 includes:

[0086] S51: Obtain several multimodal images to be segmented, preprocess the images to be segmented, and obtain the corresponding input images.

[0087] S52: Input the image to be input into the part recognition combination model. Several basic part recognition models in the part recognition combination model will sequentially detect the image to be input until the name of the part in the image to be input is determined.

[0088] S53: Input the image to be input to the segmentation base model connected to the part recognition base model that confirms the part name of the image to be input, and obtain the segmented image corresponding to the image to be segmented.

[0089] Specifically, the images obtained after processing by the part recognition basic model undergo further segmentation. Bilinear interpolation is used to upsample and learn the images to obtain dense feature maps for each image. For each dense feature map, multiple points with unknown classifications are selected, i.e., N points with the most uncertain classifications are chosen, such as points with a confidence / probability of 0.5. Then, a deep feature representation table is calculated and extracted for these N points, and the classification of each of these N points after segmentation is predicted point by point, for example, whether the point belongs to a target part or not. The above steps are repeated until the classification of each of these N points after segmentation is predicted.

[0090] When predicting the classification of each of the N segmented points, a small classifier is used to determine the category of each point, which is equivalent to using a 1x1 convolution for prediction. However, for points with confidence scores close to 1 or 0, their classification is still clear, so these points no longer need to be predicted point by point. This reduces the number of points that need to be predicted and improves the overall accuracy of the final image segmentation result. Therefore, the optimal image segmentation result is ultimately obtained.

[0091] The embodiments of this application have been described in detail above, but the content is only a preferred embodiment of this application and should not be considered as limiting the scope of this application. All equivalent changes and improvements made within the scope of this application should still fall within the patent coverage of this application.

Claims

1.A multi-modal fusion medical image segmentation method, characterized in that, Comprise: S1: determine a plurality of multi-modal original images about a plurality of target diseases of a plurality of parts; Classify the multi-modal original images to obtain a plurality of classified image sets about the parts; S2: preprocess the classified image sets about the target diseases to obtain a plurality of preprocessing data sets each including a plurality of to-be-input pictures; S3: train the CNN network structure through the preprocessing data sets to obtain a part recognition combined model taking the multi-modal original images as input items and taking the part names as output items; The step S3 comprises: S31: determine a CNN basic structure and decompose the convolution kernel of the convolution layer in the CNN basic structure to obtain a CNN network structure accordingly; S32: train a plurality of CNN network structures through a plurality of classified image sets respectively to obtain a plurality of part recognition basic models; S33: determine a scattered associative tree representing the similarity of a plurality of parts; The step S33 comprises: S331: determine the similarity between the multi-modal original pictures of a plurality of parts, and determine a part with the maximum similarity to the remaining parts as a center point; S332: draw a scattered line representing the similarity greater than the similarity threshold around the center point, and the other end of the scattered line is connected to one of the remaining parts accordingly; S334: draw a scattered line representing the similarity greater than the similarity threshold for the parts at the other end of the scattered line; S335: repeat the step S334 until the scattered associative tree representing the similarity of the parts is obtained; S34: according to the associative tree, associate the plurality of part recognition basic models to obtain a part recognition combined model accordingly; S4: train the U-Net network structure through the preprocessing data set to obtain a segmentation combined model taking the to-be-input picture as the input item and taking the segmentation picture as the output item, and connecting the part recognition combined model; The step S4 comprises: S41: according to a plurality of target diseases of each part, determine the initial training weight of the different modal original pictures; S42: determine a U-Net basic structure including an encoding sub-model and a decoding sub-model; Based on the initial training weight, introduce a multi-attention mechanism into the decoding sub-model in the U-Net basic structure to obtain a U-Net network structure; S43: train the U-Net basic structure corresponding to different initial training weights through the preprocessing data set to obtain a segmentation basic model corresponding to different parts; S44: based on the part, connect the part recognition basic model and the segmentation basic model to obtain a segmentation combined model including a plurality of segmentation basic models; S5: obtain a plurality of multi-modal to-be-segmented pictures, preprocess the to-be-segmented pictures to obtain to-be-input pictures corresponding to the to-be-segmented pictures, and input the to-be-input pictures corresponding to the to-be-segmented pictures into the part recognition combined model and the segmentation combined model in turn to obtain segmentation pictures corresponding to the to-be-segmented pictures. The step S2 comprises: 2.The multi-modal fused medical image segmentation method of claim 1, wherein, ​ S21: Normalization processing is performed on the multi-modal original images in the classified image set, and a normalized image set including a plurality of normalized pictures is obtained accordingly; S22: The normalized pictures in the normalized image set are preprocessed including cutting operation, rotation operation and elastic deformation operation, and a plurality of preprocessing data sets respectively including a plurality of to-be-input pictures are obtained accordingly. 3.The multi-modal fused medical image segmentation method of claim 1, wherein, The step S31 includes: S311: Determine the CNN basic structure; S312: Perform rank reduction decomposition processing on the convolution kernel of the convolution layer in the CNN basic structure to obtain a plurality of decomposition kernels; S313: According to the wave-shaped amplification amplitude, the plurality of decomposition kernels are amplified to obtain a plurality of amplified kernels and the corresponding CNN network structure. 4.The multi-modal fused medical image segmentation method of claim 1, wherein, The step S5 includes: S51: Obtain a plurality of multi-modal to-be-segmented pictures, and perform preprocessing on the to-be-segmented pictures to obtain to-be-input pictures corresponding to the to-be-segmented pictures; S52: Input the to-be-input pictures into the part recognition combined model, and the plurality of part recognition basic models in the part recognition combined model detect the to-be-input pictures in turn until the part name in the to-be-input pictures is determined; S53: Input the to-be-input pictures into the segmentation basic model connected with the part recognition basic model that confirms the part name of the to-be-input pictures to obtain segmented pictures corresponding to the to-be-segmented pictures.

Images