A multimodal medical image target segmentation method driven by knowledge and data fusion

Abstract

The present invention belongs to the field of medical image processing, and in particular relates to a multimodal medical image target segmentation method driven by knowledge and data fusion, the method comprising: obtaining a medical image to be detected, and performing enhancement processing on the medical image to be detected; inputting the enhanced medical image into a trained target segmentation model to obtain a segmentation result; the target segmentation model comprises a backbone network and a segmentation network; the present invention proposes a 2D / 3D data processing technology for efficient training of deep neural networks, which maximizes the retention of image features while effectively expanding the data set and improving the generalization performance of the deep neural network; the present invention also uses separable convolution and void convolution for leap-layer connection to obtain a better backbone network, which effectively expands the receptive field while having a low time complexity.

Description

Technical Field

[0001] The present invention belongs to the field of medical image processing, and in particular relates to a multimodal medical image target segmentation method driven by knowledge and data fusion. Background Art

[0002] In recent years, the incidence of malignant tumors has continued to grow and has become a public health issue of global concern. Currently, about 50% of patients with malignant tumors need to receive radiotherapy. While this treatment technology kills tumor cells, it also causes certain damage to normal tissues, leading to various complications. In order to make up for the shortcomings of traditional radiotherapy, highly conformal radiotherapy (such as intensity-modulated radiotherapy, proton therapy, and stereotactic body radiotherapy) further increases the intratumoral dose, reduces the dose to normal tissues, improves efficacy, and reduces complications. The implementation of the treatment plan depends on the precise positioning and segmentation results of the target.

[0003] At present, many traditional or AI-based image segmentation methods have been used for automatic or interactive tissue and organ segmentation tasks. For example, methods based on image grayscale information, statistical shape models, body atlases, or AI model training. However, facing the characteristics of uneven grayscale, low resolution, weak boundaries, artifact noise, or sparse individual shapes of tissues and organs in multimodal medical images, the positioning and segmentation methods based on prior knowledge or data-driven lack the study of universal and adaptive learning mechanisms. They only rely on historical observation data or big data-driven methods, and the algorithms have the risk of generalization or the model has the defect of unexplainability. The accuracy, consistency, and reproducibility of the segmentation results of the region of interest in medical images cannot be guaranteed. Among them, the algorithm based on grayscale information is sensitive to imaging artifacts and low-contrast noise; the method based on statistical shape model uses the prior knowledge of tissue and organ shape to assist single organ segmentation; the atlas-based method requires precise registration to align the target image. Due to the individual differences in the morphology of patients' organs and image artifacts, it is difficult to handle the non-smooth geometric relationship of the object in spatial distribution, size, and posture through smooth registration operations, and accurate registration operations are often not reliably guaranteed. Summary of the invention

[0004] In order to solve the problems existing in the above prior art, the present invention proposes a multimodal medical image target segmentation method based on knowledge and data fusion drive, the method comprising: obtaining a medical image to be detected, and performing enhancement processing on the medical image to be detected; inputting the enhanced medical image into a trained target segmentation model to obtain a segmentation result; the target segmentation model comprises a backbone network and a segmentation network;

[0005] The process of training the object segmentation model includes:

[0006] S1: Obtain multimodal medical imaging data from different users, and annotate the contours of tissues and organs for each multimodal medical imaging data;

[0007] S2: Perform data enhancement on the multimodal medical imaging data that divides the contours of tissues and organs to obtain a training data set;

[0008] S3: Using the human anatomical structure model library as prior knowledge, construct the optimal hierarchical structure tree for different body regions;

[0009] S4: input the data in the training data set into the corresponding optimal hierarchy tree, perform layer-order traversal from the root node of the optimal hierarchy tree, and obtain a positioning frame;

[0010] S5: Use the backbone network to extract features from the enhanced image to obtain a feature map;

[0011] S6: Using a segmentation network to perform positioning and segmentation processing on the feature map according to the positioning frame to obtain a segmentation result;

[0012] S7: Calculate the loss function of the model based on the segmentation results, continuously adjust the parameters, and complete the model training when the loss function converges.

[0013] Furthermore, the process of data enhancement for multimodal medical imaging data includes: using an adaptive data standard screening mechanism to eliminate outlier data in the multimodal medical imaging data; performing translation perturbations in the xyz direction and perspective transformations with the xyz axis as the axis according to the annotated tissue and organ contours; slicing from the sagittal plane, coronal plane and horizontal plane with the voxel to be segmented as the center; performing partition adaptive histogram equalization and intensity normalization on the above slices to obtain a training data set and a validation data set.

[0014] Furthermore, the process of removing outlier data from multimodal medical imaging data using an adaptive data standard screening mechanism includes:

[0015] Step 1: Divide the data set D into k mutually exclusive subsets of equal size, obtain data from the data set using the stratified sampling method, and distribute the collected data to each subset;

[0016] Step 2: Take the union of k-1 mutually exclusive subsets as the training set and the remaining mutually exclusive subsets as the test set, and obtain k groups of training sets and corresponding test sets.

[0017] Step 3: Input the data in the training set into the neural network model for training, and then input the data in the test set into the neural network model for forward reasoning to obtain the Dice similarity coefficient of each data;

[0018] Step 4: Use the isolation forest algorithm to remove outlier data based on the Dice similarity coefficient.

[0019] Furthermore, the expression of Dice coefficient is:

[0020]

[0021] Among them, TP is the true sample, FP is the false positive sample, and FN is the false negative sample.

[0022] Preferably, the process of constructing the optimal hierarchical structure tree for different target areas includes:

[0023] S31: dividing different tissues into regions according to the human anatomical structure model library, constructing a hierarchical structure tree to be optimized according to the divided regions, and treating the same tissue in each layer as a node in the hierarchical structure tree;

[0024] S32: starting from the first layer of the hierarchy structure tree to be optimized, performing layer-order traversal, taking nodes in the current layer as parent nodes, and nodes in the next layer as child nodes, and calculating the distance between the parent and child nodes;

[0025] S33: screening duplicate nodes in the constructed hierarchical tree according to the calculated distance between the parent and child nodes, and deleting the duplicate nodes from the hierarchical tree;

[0026] S34: Calculate the path sum of all situations of the hierarchy tree after deleting duplicate nodes, and take the hierarchy tree with the smallest path sum as the optimal hierarchy tree.

[0027] Preferably, the process of obtaining the positioning frame includes: inputting the three-dimensional image data in the training set into the optimal hierarchical structure tree, and performing a hierarchical traversal starting from the root node of the optimal hierarchical structure tree; performing a matching search inside the parent node based on the common features of the tissues and organs encoded in the node, and outputting the searched tissues and organs after slicing to obtain the positioning frame.

[0028] Preferably, the backbone network is a fully convolutional neural network, and the network structure consists of an initial layer, a jump layer connection block, a nonlinear layer, a conversion layer, and a convolution layer; the process of extracting features from the enhanced image using the backbone network includes:

[0029] S51: Positioning the enhanced image data according to the positioning frame;

[0030] S52: performing size normalization processing on the image in the positioning frame to obtain image regions to be segmented with consistent sizes;

[0031] S52: Input the image area to be segmented into the initial layer to capture the underlying details and obtain a feature map C1;

[0032] S53: Input the feature map C1 into the first convolutional layer to obtain nonlinear information and jump layer information;

[0033] S54: input the nonlinear information and the jump layer information into the first conversion layer to increase the number of channels, and perform 2 times downsampling to obtain a feature map C2;

[0034] S55: Input the feature map C2 sequentially into the second convolution layer, the second conversion layer and the third convolution layer to obtain the final feature map C3.

[0035] Preferably, the segmentation network includes a fully convolutional neural network and a fully connected classification network; and the process of using the segmentation network to segment the feature map includes:

[0036] S61: According to the position of the target area in the optimal structure hierarchy tree, dynamically select the fully convolutional neural network FCCN branch or the fully connected classification network FCNN branch to segment the feature map C3; among them, the FCCN branch is responsible for segmenting sparse and slender organs, and the FCNN branch is responsible for segmenting large-volume tissue organs;

[0037] S62: Input feature map C3 into the FCCN branch, and obtain feature map C4 after passing through the first convolution layer and the second convolution layer; input feature map C4 into the third convolution layer to obtain feature map C5; perform average pooling on feature map C5 and input it into the fully connected layer and the softmax layer to obtain the segmentation result;

[0038] S63: Input feature map C3 into the FCNN branch for convolution and maximum pooling operations to obtain feature map C4; use feature map C4 as input to perform convolution and maximum pooling operations again to obtain feature map C5; use feature map C5 as input to perform convolution and maximum pooling operations again to obtain feature map C6; finally, upsample feature map C6 to restore it to the original image size and input it into the softmax layer to obtain the segmentation result.

[0039] Preferably, the loss function expression of the model is:

[0040] The loss function of the FCNN branch is:

[0041]

[0042] Among them, y i is the label value, x i is the predicted value, n is the number of samples in a single batch;

[0043] The loss function of the FCCN branch is:

[0044] FL(p t )=-α t (1-p t )γ log(p t )

[0045] Among them, p t is the probability of predicting the t class, γ is the weight coefficient, α t is the classification weight.

[0046] Beneficial effects of the present invention:

[0047] (1) The present invention proposes a 2D / 3D data processing technology for efficient training of deep neural networks, which maximizes the retention of image features while effectively expanding the data set and improving the generalization performance of deep neural networks.

[0048] (2) The present invention makes full use of human anatomical knowledge and medical imaging data to generate an interpretable optimal structure hierarchy tree, and accurately locates the target to be segmented through the optimal structure hierarchy tree, thus producing a tissue and organ localization method with high reliability and strong robustness.

[0049] (3) The present invention utilizes separable convolution and dilated convolution to perform leap-layer connection to obtain a better backbone network, which effectively expands the receptive field while reducing the time complexity.

[0050] (4) The present invention uses the heat map of the target mask for adversarial training, which does not increase the complexity of the model and has stronger feature extraction capabilities. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 A flow chart of a multimodal medical imaging tissue and organ localization and segmentation method based on knowledge and data fusion provided by an example of the present invention;

[0052] Figure 2 A schematic diagram of outlier data of the left lung provided for an example of the present invention;

[0053] Figure 3 A schematic diagram of automatic retrieval of optimal structural relationships of tissues and organs provided by an example of the present invention;

[0054] Figure 4 The optimal structural hierarchy tree of breast tissue organs provided by the present invention example;

[0055] Figure 5 A schematic diagram of the structure of a neural network model provided for an example of the present invention;

[0056] Figure 6 A schematic diagram of the backbone network structure provided for an example of the present invention;

[0057] Figure 7 A schematic diagram of the segmentation results provided by an example of the present invention. DETAILED DESCRIPTION

[0058] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0059] A multimodal medical image target segmentation method driven by knowledge and data fusion, the method comprising: acquiring a medical image to be detected, and performing enhancement processing on the medical image to be detected; inputting the enhanced medical image into a trained target segmentation model to obtain a segmentation result; the target segmentation model comprises a backbone network and a segmentation network.

[0060] A specific implementation method of target recognition model training, such as Figure 1 As shown, the method includes:

[0061] S1: Obtain multimodal medical imaging data from different users, and annotate the contours of tissues and organs for each multimodal medical imaging data;

[0062] S2: Perform data enhancement on the multimodal medical imaging data that divides the contours of tissues and organs to obtain a training data set;

[0063] S3: Using the human anatomical structure model library as prior knowledge, construct the optimal hierarchical structure tree for different body regions;

[0064] S4: input the data in the training data set into the corresponding optimal hierarchy tree, perform layer-order traversal from the root node of the optimal hierarchy tree, and obtain a positioning frame;

[0065] S5: Use the backbone network to extract features from the enhanced image to obtain a feature map;

[0066] S6: Using a segmentation network to perform positioning and segmentation processing on the feature map according to the positioning frame to obtain a segmentation result;

[0067] S7: Calculate the loss function of the model based on the segmentation results, continuously adjust the parameters, and complete the model training when the loss function converges.

[0068] like Figure 2 As shown in the figure, the process of data enhancement for multimodal medical imaging data includes: using an adaptive data standard screening mechanism to eliminate outlier data in the multimodal medical imaging data; performing translation perturbation in the xyz direction and perspective transformation with the xyz axis as the axis according to the labeled tissue and organ contours; slicing from the sagittal plane, coronal plane and horizontal plane with the voxel to be segmented as the center; performing partition adaptive histogram equalization and intensity normalization on the above slices to obtain a training data set and a validation data set.

[0069] The process of using an adaptive data standard screening mechanism to remove outlier data from multimodal medical imaging data includes:

[0070] Step 1: Divide the data set D into k mutually exclusive subsets of equal size, obtain data from the data set using the stratified sampling method, and distribute the collected data to each subset;

[0071] Step 2: Take the union of k-1 mutually exclusive subsets as the training set and the remaining mutually exclusive subsets as the test set, and obtain k groups of training sets and corresponding test sets.

[0072] Step 3: Input the data in the training set into the neural network model for training, and then input the data in the test set into the neural network model for forward reasoning to obtain the Dice similarity coefficient of each data;

[0073] Step 4: Use the isolation forest algorithm to remove outlier data based on the Dice similarity coefficient.

[0074] The Dice coefficient is:

[0075]

[0076] Among them, TP is the true sample, FP is the false positive sample, and FN is the false negative sample.

[0077] Partition adaptive histogram equalization includes: dividing the image into real annotated areas and non-real annotated areas, and using the transformation functions calculated in the respective areas for transformation. The formula of the transformation function is:

[0078]

[0079]

[0080] Wherein, in the above formula (1) and formula (2), n is the total number of pixels of the image, L is the grayscale of the image, and n k is the gray value r k The number of pixels, p r is the gray value r k The probability density of occurrence, S k For r k Mapped to the corresponding grayscale value in the transformed image, M is the maximum grayscale value of the transformed image.

[0081] Intensity normalization specifically includes: selecting the most appropriate normalization method from Min-max, Z-score and improved logistic function according to different modal images or tissue organs, and their formulas are:

[0082]

[0083]

[0084]

[0085] Where input is the intensity of the input pixel, max is the maximum pixel intensity in the local area, min is the minimum pixel intensity in the local area, μ is the mean of the pixel intensity in the local area, σ is the standard deviation of the pixel intensity in the local area, center is the intensity of the central pixel in the local area, and scale is the scale factor.

[0086] The improved logistic function is used for normalization of the left submandibular gland, and the scale factor is set to 8. The intensity distribution obtained is more conducive to segmentation. The improved logistic function includes adding a scale factor. Different organs have different intensity distributions, and the introduction of a scale factor is more flexible.

[0087] The process of constructing the optimal hierarchical structure tree for different target areas includes: Figure 3 As shown in FIG. 1 , the optimal hierarchical structure tree of the chest region is constructed with the human anatomical structure as prior knowledge; firstly, the root node tissue organ of the chest region and the hierarchical range of the remaining tissue organs in the tree are determined according to the human anatomical knowledge to generate a hierarchical structure tree to be optimized; secondly, the hierarchical traversal starts from the root node chest skin, the current node is the parent node, the node of the next level is the child node, the distance between the parent and child nodes is calculated, and the adjacency matrix of the directed weighted graph is obtained; again, the Kruskal algorithm or the Prim algorithm is used to determine the optimal level and parent node of all nodes, and the optimal hierarchical structure tree of the chest region is obtained. The optimal hierarchical structure tree is shown in FIG. Figure 4 shown.

[0088] The shape features, grayscale features, texture features and scale features of the same tissue and organ of different individuals are extracted, and Gaussian noise of individual deviation is integrated and encoded into the corresponding nodes in the optimal structure hierarchy tree. The distance between the parent and child nodes specifically refers to the error between the child node position identified by FAM (B, G) from the parent node and the ideal position.

[0089] The process of obtaining the positioning frame according to the optimal hierarchical structure tree includes: inputting the three-dimensional image data in the training set into the optimal hierarchical structure tree, and performing a hierarchical traversal starting from the root node of the optimal hierarchical structure tree; performing a matching search inside the parent node according to the common features of the tissues and organs encoded in the node, and outputting the searched tissues and organs after slicing to obtain the positioning frame.

[0090] A flexible multi-branch segmentation network is constructed based on the fully connected classification network (FCCN) and the fully convolutional neural network (FCNN), such as Figure 5As shown in the figure, the backbone network specifically includes a residual module, pooling, downsampling, a fully connected layer and a Softmax layer; the segmentation head consists of two branches, FCCN and FCNN, and sparse and slender organs and large-volume tissue organs are segmented by the FCCN branch and the FCNN branch respectively.

[0091] Specifically, Figure 6 As shown in the figure, the backbone network is a fully convolutional neural network, which consists of an initial layer, a jump-layer connection block, a nonlinear layer, a conversion layer, and a convolutional layer. The process of extracting features from the enhanced image using the backbone network includes:

[0092] S51: Positioning the enhanced image data according to the positioning frame;

[0093] S52: performing size normalization processing on the image in the positioning frame to obtain image regions to be segmented with consistent sizes;

[0094] S52: Input the image area to be segmented into the initial layer to capture the underlying details and obtain a feature map C1;

[0095] S53: Input the feature map C1 into the first convolutional layer to obtain nonlinear information and jump layer information;

[0096] S54: input the nonlinear information and the jump layer information into the first conversion layer to increase the number of channels, and perform 2 times downsampling to obtain a feature map C2;

[0097] S55: Input the feature map C2 sequentially into the second convolution layer, the second conversion layer and the third convolution layer to obtain the final feature map C3.

[0098] The segmentation network includes a fully convolutional neural network and a fully connected classification network. The process of segmenting the feature map using the segmentation network includes:

[0099] S61: According to the position of the target area in the optimal structure hierarchy tree, dynamically select the fully convolutional neural network FCCN branch or the fully connected classification network FCNN branch to segment the feature map C3; among them, the FCCN branch is responsible for segmenting sparse and slender organs, and the FCNN branch is responsible for segmenting large-volume tissue organs;

[0100] S62: Input feature map C3 into the FCCN branch, and obtain feature map C4 after passing through the first convolution layer and the second convolution layer; input feature map C4 into the third convolution layer to obtain feature map C5; perform average pooling on feature map C5 and input it into the fully connected layer and the softmax layer to obtain the segmentation result;

[0101] S63: Input feature map C3 to the FCNN branch for convolution and maximum pooling operations to obtain feature map C4; use feature map C4 as input to perform convolution and maximum pooling operations again to obtain feature map C5; use feature map C5 as input to perform convolution and maximum pooling operations again to obtain feature map C6; finally, upsample feature map C6 to restore it to the original image size and input it to the softmax layer to obtain the segmentation result. The segmentation result is as follows: Figure 7 shown.

[0102] The loss function of the FCNN branch is:

[0103]

[0104] Among them, y i is the label value, σ(x i ) is the predicted value.

[0105] The loss function of the FCCN branch is:

[0106] FL(p t )=-α t (1-p t ) γ log(p t )

[0107] Among them, p t is the probability of predicting the t class, γ is the weight coefficient, α t is the classification weight.

[0108] A specific implementation method for training a target segmentation model, the method steps comprising:

[0109] Step (1): Input the preprocessed training set data into the neural network for training until the accuracy on the validation set reaches a peak;

[0110] Step (2): Input all the data in the training set into the model for forward reasoning, and use the gradient-based category heat map (Grad-CAM) method to obtain the probability mask (one-to-one correspondence with the image);

[0111] Step (3): Fine-tune the network model, multiply the feature map output by the backbone network by the probability mask obtained in step 6 point by point, and then input it into the segmentation head for training. Repeat steps (2) and (3) until the most discriminative area basically covers the true annotation area;

[0112] Among them, the method based on gradient category heat map (Grad-CAM) specifically includes:

[0113]

[0114]

[0115]

[0116] Among them, L c is the category heat map, A k is the data of channel k in feature layer A, Z is the number of pixels in feature layer A, is a single pixel or output mask, A k The data at (i, j) in y ij is the prediction score at (i, j), L p is the probability mask, For L c The value at (i, j) in .

[0117] Read the 3D image data of the chest, start from the root node of the optimal hierarchical structure tree to traverse the layers, input the model for forward reasoning, and obtain the target mask. According to the common features of the tissues and organs encoded in the node, search for matching inside the parent node, and output the searched tissues and organs after slicing.

[0118] The above embodiments further illustrate the purpose, technical solutions and advantages of the present invention in detail. It should be understood that the above embodiments are only preferred implementation modes of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made to the present invention within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A multimodal medical image target segmentation method based on knowledge and data fusion, characterized in that: The method comprises: obtaining a medical image to be detected, and performing enhancement processing on the medical image to be detected; inputting the enhanced medical image into a trained target segmentation model to obtain a segmentation result; the target segmentation model comprises a backbone network and a segmentation network; The process of training the object segmentation model includes: S1: Obtain multimodal medical imaging data from different users, and annotate the contours of tissues and organs for each multimodal medical imaging data; S2: Perform data enhancement on the multimodal medical imaging data that divides the contours of tissues and organs to obtain a training data set; S3: Using the human anatomical structure model library as prior knowledge, construct the optimal hierarchical structure tree for different body regions; S4: input the data in the training data set into the corresponding optimal hierarchy tree, perform layer-order traversal from the root node of the optimal hierarchy tree, and obtain a positioning frame; S5: The backbone network is used to extract features from the enhanced image to obtain a feature map; the backbone network is a fully convolutional neural network, and the network structure consists of an initial layer, a jump layer connection block, a nonlinear layer, a conversion layer, and a convolution layer; the process of extracting features from the enhanced image using the backbone network includes: S51: Positioning the enhanced image data according to the positioning frame; S52: performing size normalization processing on the image in the positioning frame to obtain image regions to be segmented with consistent sizes; S52: Input the image area to be segmented into the initial layer to capture the underlying details and obtain a feature map C1; S53: Input the feature map C1 into the first convolutional layer to obtain nonlinear information and jump layer information; S54: input the nonlinear information and the jump layer information into the first conversion layer to increase the number of channels, and perform 2 times downsampling to obtain a feature map C2; S55: input the feature map C2 into the second convolution layer, the second conversion layer and the third convolution layer in sequence to obtain the final feature map C3; S6: Using a segmentation network to perform positioning and segmentation processing on the feature map according to the positioning frame to obtain a segmentation result; the segmentation network includes a fully convolutional neural network and a fully connected classification network; the process of using the segmentation network to segment the feature map includes: S61: According to the position of the target area in the optimal structure hierarchy tree, dynamically select the fully convolutional neural network FCCN branch or the fully connected classification network FCNN branch to segment the feature map C3; among them, the FCCN branch is responsible for segmenting sparse and slender organs, and the FCNN branch is responsible for segmenting large-volume tissue organs; S62: Input feature map C3 into the FCCN branch, and obtain feature map C4 after passing through the first convolution layer and the second convolution layer; input feature map C4 into the third convolution layer to obtain feature map C5; perform average pooling on feature map C5 and input it into the fully connected layer and the softmax layer to obtain the segmentation result; S63: Input feature map C3 to the FCNN branch for convolution and maximum pooling operations to obtain feature map C4; use feature map C4 as input to perform convolution and maximum pooling operations again to obtain feature map C5; use feature map C5 as input to perform convolution and maximum pooling operations again to obtain feature map C6; finally, upsample feature map C6 to restore it to the original image size and input it to the softmax layer to obtain the segmentation result; S7: Calculate the loss function of the model based on the segmentation results, continuously adjust the parameters, and complete the model training when the loss function converges.

2. According to claim 1, a multimodal medical image target segmentation method based on knowledge and data fusion driven, characterized in that: The process of data enhancement for multimodal medical imaging data includes: using an adaptive data standard screening mechanism to eliminate outlier data in the multimodal medical imaging data; performing translation perturbations in the xyz direction and perspective transformations with the xyz axis as the axis according to the annotated tissue and organ contours; slicing from the sagittal plane, coronal plane and horizontal plane with the voxel to be segmented as the center; performing partition adaptive histogram equalization and intensity normalization on the above slices to obtain a training data set.

3. According to claim 2, a multimodal medical image target segmentation method based on knowledge and data fusion driven, characterized in that: The process of using an adaptive data standard screening mechanism to remove outlier data from multimodal medical imaging data includes: Step 1: Divide the data set D into k mutually exclusive subsets of equal size, obtain data from the data set using the stratified sampling method, and distribute the collected data to each subset; Step 2: Take the union of k-1 mutually exclusive subsets as the training set and the remaining mutually exclusive subsets as the test set, and obtain k sets of training sets and corresponding test sets; Step 3: Input the data in the training set into the neural network model for training, and then input the data in the test set into the neural network model for forward reasoning to obtain the Dice similarity coefficient of each data; Step 4: Use the isolation forest algorithm to remove outlier data based on the Dice similarity coefficient.

4. According to claim 3, a multimodal medical image target segmentation method based on knowledge and data fusion driven, characterized in that: The expression of Dice coefficient is: Among them, TP is the true sample, FP is the false positive sample, and FN is the false negative sample.

5. According to claim 1, a multimodal medical image target segmentation method based on knowledge and data fusion driven, characterized in that: The process of constructing the optimal hierarchical structure tree for different target areas includes: S31: dividing different tissues into regions according to the human anatomical structure model library, constructing a hierarchical structure tree to be optimized according to the divided regions, and treating the same tissue in each layer as a node in the hierarchical structure tree; S32: starting from the first layer of the hierarchy structure tree to be optimized, performing layer-order traversal, taking nodes in the current layer as parent nodes, and nodes in the next layer as child nodes, and calculating the distance between the parent and child nodes; S33: screening duplicate nodes in the constructed hierarchical tree according to the calculated distance between the parent and child nodes, and deleting the duplicate nodes from the hierarchical tree; S34: Calculate the path sum of all situations of the hierarchy tree after deleting duplicate nodes, and take the hierarchy tree with the smallest path sum as the optimal hierarchy tree.

6. The method for multimodal medical image target segmentation based on knowledge and data fusion drive according to claim 1, characterized in that: The process of obtaining the positioning frame includes: inputting the three-dimensional image data in the training set into the optimal hierarchical structure tree, and performing a hierarchical traversal starting from the root node of the optimal hierarchical structure tree; performing a matching search inside the parent node according to the common features of the tissues and organs encoded in the node, and outputting the searched tissues and organs after slicing to obtain the positioning frame.

7. The method for multimodal medical image target segmentation based on knowledge and data fusion drive according to claim 1, characterized in that: The loss function expression of the model is: The loss function of the FCNN branch is: Among them, y i is the label value, x i is the predicted value, n is the number of samples in a single batch; The loss function of the FCCN branch is: FL(p t )=-a t (1-p t ) γ log(p t ) Among them, p t is the probability of predicting the t class, γ is the weight coefficient, α t is the classification weight.

Images