A multimodal interactive medical image segmentation method and apparatus

Abstract

The application discloses a kind of multimodal interactive medical image segmentation method and device.Therein, the method includes: reading multimodal medical image input data, obtaining the preprocessed data of target size;Training set sample is input to multimodal rough segmentation network, and the segmentation probability graph is obtained;Obtain the error correction suggestion to segmentation probability graph, if it does not meet preset standard, then according to error information in error correction suggestion, the coordinates of error point are determined, and error information is mapped as multimodal distance measurement graph according to the coordinates of error point;Multimodal distance measurement graph, multimodal medical image and segmentation probability graph are spliced, and are sent into multimodal interactive medical image segmentation network, and the new segmentation probability graph is obtained;The new segmentation probability graph obtained by iteration exceeds maximum sequence length or segmentation probability graph meets standard.The scheme can achieve more efficient and accurate segmentation, and the precise segmentation of medical image is completed through human-computer interaction to achieve clinical use standard.

Description

Technical Field

[0001] This invention relates to the fields of machine learning and image processing technology, and in particular to a multimodal interactive medical image segmentation method and apparatus. Background Technology

[0002] In recent years, with the rapid development of medical technology and the dramatic increase in patient data in hospitals, a large amount of medical imaging data urgently needs processing. Manual identification and analysis places excessive demands on doctors and relies too heavily on their condition and experience, failing to meet current data analysis needs. For example, diagnosing a patient's tumor often requires collecting medical image data from multiple modalities, and human experts need to comprehensively utilize this multimodal medical image data during the diagnostic process, placing extremely high demands on manual identification. Summary of the Invention

[0003] This application provides a multimodal interactive medical image segmentation method and apparatus, which can make full use of multimodal medical images and achieve expert guidance and correction of model segmentation results through interactive segmentation, thereby achieving more efficient and accurate segmentation. The accurate segmentation of medical images is completed through human-computer interaction to meet clinical use standards.

[0004] In a first aspect, this application provides a multimodal interactive medical image segmentation method, the method comprising:

[0005] Read multimodal medical image input data and determine the cropping range according to the pre-labeled target category to crop the image and obtain preprocessed data of the target size;

[0006] The preprocessed data is concatenated along the channel dimension, and then divided into a training set and a test set according to a preset ratio.

[0007] The training set samples are input into a multimodal coarse segmentation network to obtain a segmentation probability map;

[0008] Obtain error correction suggestions for the segmentation probability map; if the segmentation probability map meets the preset criteria, output the segmentation probability map as the final segmentation result.

[0009] If the segmentation probability map does not meet the preset standard, then the coordinates of the error points are determined according to the error information in the error correction suggestions, and the error information is mapped into a multimodal distance metric map according to the coordinates of the error points;

[0010] The multimodal distance metric map, multimodal medical image, and segmentation probability map are stitched together along the channel dimension, and the stitching result is fed into the multimodal interactive medical image segmentation network to obtain a new segmentation probability map.

[0011] Obtain error correction suggestions for the new segmentation probability map until the new segmentation probability map obtained through iteration exceeds the maximum sequence length or the segmentation probability map meets the standard.

[0012] Furthermore, after inputting the training set samples into a multimodal coarse segmentation network to obtain a segmentation probability map, the method further includes:

[0013] By combining the segmentation probability map and the ground truth of the training set samples, the cross-entropy loss is calculated and optimized using the gradient descent method.

[0014] Furthermore, after feeding the stitched result into a multimodal interactive medical image segmentation network to obtain a new segmentation probability map, the method further includes:

[0015] By combining the new segmentation probability map and the ground truth of the training set samples, the cross-entropy loss is calculated and optimized using gradient descent.

[0016] Furthermore, the method also includes:

[0017] The tumor is classified hierarchically according to different levels of lesions, and the changes in the tumor are modeled. The specific calculation formula is as follows:

[0018]

[0019]

[0020] Among them, L new For the new label, L BG For the background class in the original tag, L ED For the edema category in the original label, L NET For the non-enhancing tumor category in the original label, L ET For the enhanced tumor category in the original label, L ′ BG For the transformed background class, L ′ WT For the transformed total tumor category, L ′ TC For the transformed tumor core category, L ′ ET H represents the enhanced tumor category after transformation, where H is the transformation function for each pixel, and x represents the value of that pixel after matrix operations.

[0021] Furthermore, after obtaining the preprocessed data of the target size, the method further includes:

[0022] Data augmentation is performed on the preprocessed data of the target size to obtain an augmented dataset; wherein the data augmentation includes flipping along an arbitrary one-dimensional spatial coordinate axis.

[0023] Furthermore, the backbone structure of the multimodal coarse segmentation network is a three-dimensional U-Net network, which includes a multimodal aggregation module and a hierarchical prediction module.

[0024] The multimodal coarse segmentation network specifically includes: a multimodal aggregation module, three downsampling modules, one bridging module, three upsampling modules, and a hierarchical prediction module;

[0025] The multimodal aggregation module includes an initialization submodule and a channel weighting submodule;

[0026] The initialization submodule contains an IN operator and a convolutional layer with a kernel size of 1×1×1, a stride of 1, and padding of 0.

[0027] The channel weighting submodule includes a pooling layer, two fully connected layers, and a pixel-wise multiplication operator; the pooling layer uses max pooling, and the pooling layer kernel size is the image size; the fully connected layer has 16 neurons; the pixel-wise multiplication operator is used to assign the weight values ​​after the fully connected layer to each channel of the feature.

[0028] The downsampling module includes two convolutional layers and one pooling layer. The convolutional layer has a kernel size of 3×3×3, a stride of 1, and padding of 1. The pooling layer has a kernel size of 2×2×2 and a stride of 2.

[0029] The bridging module includes two convolutional layers, each with a kernel size of 3×3×3, a stride of 1, and padding of 1.

[0030] The upsampling module includes one deconvolution layer and two convolutional layers. The deconvolution operation has a kernel size of 3×3×3, a stride of 2, and padding of 1. The convolutional layers also have a kernel size of 3×3×3, a stride of 1, and padding of 1.

[0031] The hierarchical prediction module includes three sub-prediction modules, each of which includes four convolutional layers. There is an activation operator after every two convolutional layers. Except for the last convolutional layer, the kernel size is 3×3×3, the stride is 1, and the padding is 1. The first activation operator in each sub-prediction module is ReLU, and the second activation operator is Sigmoid.

[0032] Furthermore, the multimodal interactive medical image segmentation network specifically includes: a multimodal aggregation module, five basic blocks, and a hierarchical prediction module;

[0033] In the aforementioned basic blocks, the first two basic blocks contain two dilated convolutions, and the last three basic blocks contain three dilated convolutions. The dilated convolutions within the same basic block have the same dilation rate, a kernel size of 3, a stride of 1, and padding equal to the dilation rate of the basic block plus 1. The dilation rates of each basic block are 0, 1, 2, 3, and 4, respectively.

[0034] Furthermore, combining the new segmentation probability map and the ground truth of the training set samples, the cross-entropy loss is calculated, and the cross-entropy loss is optimized using gradient descent, including:

[0035] The binary cross-entropy is calculated using the following formula:

[0036]

[0037]

[0038] Among them, L total Let C be the overall binary cross-entropy loss, C be the number of lesion grades, L be the true label, and P be the predicted probability. BCE Here is the formula for calculating binary cross-entropy loss, where Q is the label of a lesion of any grade. This is the prediction result for this sample.

[0039] Furthermore, experts corrected errors based on the existing segmentation results, including:

[0040] The system identifies mouse clicks in error areas, treating each click as a foreground point. It then calculates the geodesic distance between each pixel in the image and the foreground point, using the following formula:

[0041]

[0042] Where, x i and x j These are two pixels, P i,j Let be the set of all paths between two points, and p be a feasible path parameterized by s. It is a unit vector tangent to the direction of travel.

[0043] Secondly, this application provides a multimodal interactive medical image segmentation device, the device comprising:

[0044] The preprocessing data determination unit is used to read multimodal medical image input data and determine the cropping range according to the pre-labeled target category in order to crop the image and obtain preprocessed data of the target size;

[0045] The training set partitioning unit is used to concatenate the preprocessed data along the channel dimension and divide the preprocessed data into a training set and a test set according to a preset ratio.

[0046] The segmentation probability map output unit is used to input training set samples into the multimodal coarse segmentation network to obtain a segmentation probability map;

[0047] The final segmentation result determination unit is used to obtain error correction suggestions for the segmentation probability map. If the segmentation probability map meets the preset standard, the segmentation probability map is output as the final segmentation result.

[0048] A multimodal distance metric map mapping unit is used to determine the coordinates of the error points based on the error information in the error correction suggestions if the segmentation probability map does not meet the preset standard, and to map the error information into a multimodal distance metric map based on the coordinates of the error points.

[0049] The new segmentation probability map output unit is used to stitch the multimodal distance metric map, multimodal medical image and segmentation probability map in the channel dimension, and send the stitching result into the multimodal interactive medical image segmentation network to obtain a new segmentation probability map;

[0050] An iterative statistical unit is used to obtain error correction suggestions for the new segmentation probability map until the new segmentation probability map obtained through iteration exceeds the maximum sequence length or the segmentation probability map meets the standard.

[0051] The technical solution provided in this application, a multimodal interactive medical image segmentation method and device based on hierarchical multi-label prediction, can make full use of multimodal medical images and achieve expert guidance and correction of model segmentation results through interactive segmentation, thereby achieving more efficient and accurate segmentation. It completes the precise segmentation of medical images through human-computer interaction to meet clinical use standards. Attached Figure Description

[0052] Figure 1 This is a flowchart of the multimodal interactive medical image segmentation method provided in Embodiment 1 of this application;

[0053] Figure 2 A schematic diagram of the multimodal aggregation module provided in Embodiment 1 of this application;

[0054] Figure 3 A schematic diagram of the multimodal hierarchical prediction module provided in Embodiment 1 of this application;

[0055] Figure 4 A schematic diagram of the multimodal interactive medical image segmentation network provided in Embodiment 1 of this application;

[0056] Figure 5This is a schematic diagram of the overall architecture provided in Embodiment 2 of this application;

[0057] Figure 6 This is a structural block diagram of the multimodal interactive medical image segmentation device provided in Embodiment 3 of the present invention. Detailed Implementation

[0058] The present application will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the application and not intended to limit it. Furthermore, it should be noted that, for ease of description, the accompanying drawings show only the parts relevant to the present application, not the entire structure.

[0059] Before discussing the exemplary embodiments in more detail, it should be noted that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although the flowcharts describe the steps as sequential processes, many of these steps can be performed in parallel, concurrently, or simultaneously. Furthermore, the order of the steps can be rearranged. The process can be terminated when its operation is complete, but may also have additional steps not included in the figures. The process can correspond to a method, function, procedure, subroutine, subroutine, etc.

[0060] Example 1

[0061] Figure 1 This is a flowchart of the multimodal interactive medical image segmentation method provided in Embodiment 1 of this application. This embodiment can be applied to scenarios of interactive segmentation of lesion images. The method can be executed by the multimodal interactive medical image segmentation device provided in this embodiment of the application. The device can be implemented by software and / or hardware and can be integrated into an electronic device.

[0062] like Figure 1 As shown, the multimodal interactive medical image segmentation method includes:

[0063] S110 reads multimodal medical image input data and determines the cropping range according to the pre-labeled target category to crop the image and obtain preprocessed data of the target size.

[0064] This approach requires remodeling the multi-classification problem in tumor segmentation, treating it as a hierarchical multi-label prediction task, and using the idea of ​​multi-label classification to solve the problem.

[0065] Optionally, in this solution, the method further includes:

[0066] The tumor is classified hierarchically according to different levels of lesions, and the changes in the tumor are modeled. The specific calculation formula is as follows:

[0067]

[0068]

[0069] Among them, L new For the new label, L BG For the background class in the original tag, L ED For the edema category in the original label, L NET For the non-enhancing tumor category in the original label, L ET For the enhanced tumor category in the original label, L′ BG For the transformed background class, L′ WT For the transformed total tumor category, L′ TC For the transformed tumor core category, L′ ET H represents the enhanced tumor category after transformation, where H is the transformation function for each pixel, and x represents the value of that pixel after matrix operations.

[0070] After this label transformation, the original multi-classification problem will be transformed into a hierarchical multi-label prediction problem, which is more in line with the actual physical meaning.

[0071] Specifically, the ratio of the training set to the validation set is 4:1. In order to enable medical images to better adapt to the unified network parameters and to remove unnecessary noise points so that the ratio of the target to the background can be increased, this invention performs data preprocessing on all medical data when reading the input data.

[0072] The input image is read and preprocessed. Specifically, six faces of the bounding box (3D single-channel image data) of the target category from the manually labeled data are extracted. These six faces are then randomly expanded outwards by 1 to 11 voxels, and cropped using the new bounding box to obtain the target size. The image is then scaled to 96×96×96 using trilinear interpolation. This preprocessing method increases the target's proportion and makes the image size relatively uniform, suitable for consistent network parameters.

[0073] The six faces of the bounding box can be the six faces of a cube, namely the top, bottom, left, right, front, and back faces.

[0074] In this scheme, the sample images can be enhanced after preprocessing.

[0075] S120, the preprocessed data is spliced ​​together along the channel dimension, and the preprocessed data is divided into a training set and a test set according to a preset ratio.

[0076] In this context, the channel dimension can be the channel of each slice in a medical image. For example, a CT image consists of 16 images obtained by taking 16 slices of the lesion, and the channel dimension can be 16 slice channels.

[0077] In this embodiment, optionally, after obtaining the preprocessed data of the target size, the method further includes:

[0078] Data augmentation is performed on the preprocessed data of the target size to obtain an augmented dataset; wherein the data augmentation includes flipping along an arbitrary one-dimensional spatial coordinate axis.

[0079] In this approach, channel dimensions and standardization can be added to increase the diversity of the dataset and improve the generalization performance of the model, while keeping the data in the same distribution.

[0080] The specific implementation method for data augmentation here is as follows:

[0081] Choose one of the following options with uniform probability: flip along the x-axis, flip along the y-axis, flip along the z-axis, or do not flip.

[0082] The method for standardizing image data is Z-Score standardization, which transforms the enhanced image distribution into a standard normal distribution. The specific calculation method is as follows:

[0083]

[0084] Where x is the input image, x * The output image is defined as follows: μ is the mean of the pixel values ​​in the entire image, δ is the variance of the pixel values ​​in the entire image, and x is the result. * It follows a standard normal distribution.

[0085] After obtaining the above operation results, the preprocessed data needs to be divided into training set and test set according to a preset ratio of 4:1.

[0086] S130: Input the training set samples into the multimodal coarse segmentation network to obtain the segmentation probability map.

[0087] The backbone structure of the multimodal coarse segmentation network is a three-dimensional U-Net network, with the addition of a multimodal aggregation module and a hierarchical prediction module.

[0088] In this embodiment, optionally, the backbone structure of the multimodal coarse segmentation network is a three-dimensional U-Net network, which includes a multimodal aggregation module and a hierarchical prediction module.

[0089] The multimodal coarse segmentation network specifically includes: a multimodal aggregation module, three downsampling modules, one bridging module, three upsampling modules, and a hierarchical prediction module;

[0090] The multimodal aggregation module includes an initialization submodule and a channel weighting submodule;

[0091] The initialization submodule contains an IN operator and a convolutional layer with a kernel size of 1×1×1, a stride of 1, and padding of 0.

[0092] The channel weighting submodule includes a pooling layer, two fully connected layers, and a pixel-wise multiplication operator; the pooling layer uses max pooling, and the pooling layer kernel size is the image size; the fully connected layer has 16 neurons; the pixel-wise multiplication operator is used to assign the weight values ​​after the fully connected layer to each channel of the feature.

[0093] The downsampling module includes two convolutional layers and one pooling layer. The convolutional layer has a kernel size of 3×3×3, a stride of 1, and padding of 1. The pooling layer has a kernel size of 2×2×2 and a stride of 2.

[0094] The bridging module includes two convolutional layers, each with a kernel size of 3×3×3, a stride of 1, and padding of 1.

[0095] The upsampling module includes one deconvolution layer and two convolutional layers. The deconvolution operation has a kernel size of 3×3×3, a stride of 2, and padding of 1. The convolutional layers also have a kernel size of 3×3×3, a stride of 1, and padding of 1.

[0096] The hierarchical prediction module includes three sub-prediction modules, each of which includes four convolutional layers. There is an activation operator after every two convolutional layers. Except for the last convolutional layer, the kernel size is 3×3×3, the convolution stride is 1, and the padding is 1. The first activation operator in each sub-prediction module is ReLU, and the second activation operator is Sigmoid.

[0097] Figure 2 This is a schematic diagram of the multimodal aggregation module provided in Embodiment 1 of this application, as shown below. Figure 2 As shown, it includes an initialization submodule and a channel weighting submodule. The initialization submodule contains an IN operator and a convolutional layer with a kernel size of 1×1×1, a stride of 1, and padding of 0. The channel weighting submodule contains a pooling layer, two fully connected layers, and a pixel-wise multiplication operator. The pooling layer uses max pooling with a kernel size equal to the image size. The fully connected layer has 16 neurons. The pixel-wise multiplication operator is used to assign the weight values ​​after the fully connected layer to each channel of the feature.

[0098] Figure 3 This is a schematic diagram of the multimodal hierarchical prediction module provided in Embodiment 1 of this application, as shown below. Figure 3As shown, it contains three sub-prediction modules to output predicted labels at different levels. Each sub-prediction module includes four convolutional layers, with an activation operator after every two convolutional layers. Except for the last convolutional layer, the kernel size is 3×3×3, the convolution stride is 1, and the padding is 1. The first activation operator in each sub-prediction module is ReLU, and the second activation operator is Sigmoid.

[0099] In this implementation example, the multimodal coarse segmentation network comprises three downsampling modules for feature encoding and reducing the number of computational parameters, a bridging module for calculating the deepest image features and bridging the upsampling and downsampling processes, three upsampling modules for feature decoding, and a final aggregation convolution for the output. The image is input into this network to obtain the output probability map.

[0100] In this implementation example, the data is first input into the multimodal aggregation module, then the result is input into the downsampling module, retaining the results before each of the three pooling steps. The output of the last downsampling module is then input into the bridging module, and its output is input into the upsampling module. The inputs from the previous downsampling modules are sequentially concatenated to the output of the deconvolution to obtain the features, which are then input into the hierarchical prediction module to obtain the final required probability map P. In the labeled data, the cross-entropy loss is calculated using the obtained probability map and the ground truth, and this loss function is used as the final loss function.

[0101] In one feasible embodiment, optionally, after feeding the stitching result into a multimodal interactive medical image segmentation network to obtain a new segmentation probability map, the method further includes:

[0102] By combining the new segmentation probability map and the ground truth of the training set samples, the cross-entropy loss is calculated and optimized using gradient descent.

[0103] Specifically, based on the segmentation probability map and the ground truth of the samples, the binary cross-entropy loss is calculated, including:

[0104] The binary cross-entropy loss is calculated using the following formula:

[0105]

[0106]

[0107] Among them, L total Let C be the total loss, L be the number of lesion grades, P be the predicted probability, and L be the total loss. BCE Here is the formula for calculating binary cross-entropy loss, where Q is the label of a lesion of any grade. This is the prediction result for this sample.

[0108] S140, obtain error correction suggestions for the segmentation probability map. If the segmentation probability map meets the preset standard, output the segmentation probability map as the final segmentation result.

[0109] The segmentation probability map can be fed back to medical experts, who can then interact by clicking within the error area. The geodesic distance between each voxel in the original image and the interaction point is calculated, and the generated distance metric map, segmentation probability map, and original multimodal image are stitched together along the channel dimension.

[0110] Here, if the expert analysis indicates that the segmentation probability map meets the preset criteria, the segmentation probability map will be output as the final segmentation result.

[0111] S150, if the segmentation probability map does not meet the preset standard, then the coordinates of the error points are determined according to the error information in the error correction suggestion, and the error information is mapped into a multimodal distance metric map according to the coordinates of the error points.

[0112] Here, if the expert analysis indicates that the segmentation probability map does not meet the preset standards, then the expert needs to provide error correction suggestions, which must include error information. The error information may include the coordinates of the error points and the type of error.

[0113] Optionally, in this technical solution, the coordinates of the error point are determined based on the error information in the error correction suggestion, and the error information is mapped into a multimodal distance metric map based on the coordinates of the error point, including:

[0114] The system identifies mouse clicks in error areas, treating each click as a foreground point. It then calculates the geodesic distance between each pixel in the image and the foreground point, using the following formula:

[0115]

[0116] Where, x i and x j These are two pixels, P i,j Let be the set of all paths between two points, and p be a feasible path parameterized by s. It is a unit vector tangent to the direction of travel.

[0117] Specifically, the expert's interaction method involves clicking the mouse in the error area. Each click is considered a foreground point, and the geodesic distance between each pixel in the image and the foreground point is calculated.

[0118] In this solution, optionally, the multimodal interactive medical image segmentation network specifically includes: a multimodal aggregation module, five basic blocks, and a hierarchical prediction module;

[0119] In the aforementioned basic blocks, the first two basic blocks contain two dilated convolutions, and the last three basic blocks contain three dilated convolutions. The dilated convolutions within the same basic block have the same dilation rate, a kernel size of 3, a stride of 1, and padding equal to the dilation rate of the basic block plus 1. The dilation rates of each basic block are 0, 1, 2, 3, and 4, respectively.

[0120] The backbone network of the multimodal interactive medical image segmentation network is a three-dimensional P-Net, consisting of a multimodal aggregation module, five basic blocks, and a hierarchical prediction module.

[0121] The multimodal aggregation module and hierarchical prediction module are the same as the two modules mentioned above, and will not be described again here.

[0122] Figure 4 This is a schematic diagram of the multimodal interactive medical image segmentation network provided in Embodiment 1 of this application, as shown below. Figure 4 As shown, the network comprises a multimodal data aggregation modality, five basic modules, and a hierarchical prediction module. The basic sub-modules are dilated convolution modules. The first two sub-modules contain two dilated convolutions, and the last three sub-modules contain three dilated convolutions. All dilated convolutions within each sub-module use the same dilation rate, with a kernel size of 3, a stride of 1, and padding equal to the kernel dilation rate plus 1. The dilation rates for each basic block are 0, 1, 2, 3, and 4, respectively. The outputs of each sub-module are concatenated and merged into the input features of the hierarchical prediction module, thereby comprehensively utilizing image features from different semantic levels.

[0123] S160, the multimodal distance metric map, multimodal medical image and segmentation probability map are stitched together in the channel dimension, and the stitching result is sent into the multimodal interactive medical image segmentation network to obtain a new segmentation probability map.

[0124] The process involves inputting samples from the test set into a multimodal coarse segmentation network to obtain coarse segmentation results. These results are then fed back to medical experts to obtain interactive error correction information. Subsequently, the original image, interactive error correction information, and coarse segmentation results are stitched together on a channel and input into a multimodal interactive segmentation network to obtain the final segmentation result. This process of feedback, interaction, and updating of segmentation results iterates continuously until the maximum sequence length is exceeded or the result is approved by the experts.

[0125] S170, obtain error correction suggestions for the new segmentation probability map until the new segmentation probability map obtained through iteration exceeds the maximum sequence length or the segmentation probability map meets the standard.

[0126] To enhance the comprehensive utilization of multimodal medical image data, this scheme designs a multimodal data aggregation module to simultaneously integrate image features from different modalities. Simultaneously, to address the limitations of binary interaction methods, this scheme, based on the characteristics of lesion segmentation, remodels the multi-classification problem into a hierarchical multi-label prediction problem and designs a hierarchical prediction module to improve the model's segmentation performance. Unlike conventional multimodal interactive medical image segmentation methods, this scheme allows for the simultaneous optimization of multiple classification targets in a single interaction, while conventional methods require building multiple interactive segmentation models and processing each segmentation target separately through multiple interactions.

[0127] This invention, by introducing multimodal data and interactive methods, achieves a Dice of 90.62% for whole tumor (WT) segmentation on the Brain Tumor Segmentation Challenge 2019 (BraTS2019) dataset, a 5.43% improvement compared to the results obtained using only 3D U-Net on the same data. For tumor core (TC) segmentation, the proposed method achieves a Dice of 76.92%, a 14.41% improvement compared to the results obtained using only 3D U-Net on the same data. For enhancing tumor (ET) segmentation, the proposed method achieves a Dice of 66.42%, a 6.78% improvement compared to the results obtained using only 3D U-Net on the same data. Compared to the results of using the interactive segmentation method DeepIGeos directly on multi-class classification problems, this approach improves segmentation performance by 4.5% for the entire tumor class, 14.49% for the core tumor class, and 8.54% for the enhanced tumor class, with the same number of interaction rounds. BraTS2019 is a medical dataset for brain tumors, and Dice is a common detection metric.

[0128] The technical solution provided in this application, by remodeling the segmentation problem, designing a multimodal coarse segmentation network, and a multimodal interactive medical image segmentation network, completes the construction of the overall model, thereby obtaining a systematic multimodal interactive medical image segmentation method for multimodal medical image data, achieving the goal of quickly and accurately completing multimodal interactive medical image segmentation.

[0129] Example 2

[0130] This embodiment is a preferred embodiment based on the above embodiments. Figure 5 This is a schematic diagram of the overall architecture provided in Embodiment 2 of this application. The multimodal interactive medical image segmentation method based on hierarchical multi-label prediction provided by this solution includes the following steps:

[0131] Step 1: Remodel the multi-classification problem in tumor segmentation into a hierarchical multi-label prediction problem. Classify lesions of different levels in the tumor segmentation problem hierarchically and model the tumor's change process. The specific calculation formula is as follows:

[0132]

[0133]

[0134] Among them, L new For the new label, L BG For the background class in the original tag, L ED For the edema category L in the original label NET For the non-enhancing tumor category in the original label, L ET For the enhanced tumor in the original label, L′ BG For the transformed background class, L′ WT For the transformed whole tumor class (WT), L′ TC For the transformed tumor core class, L′ ET For the transformed enhanced tumor class, H is the transformation function for each pixel, and x represents the value of that pixel after matrix operation.

[0135] Step 2: When reading the input image, preprocessing is required. The preprocessing operation is as follows:

[0136] Extract the six faces (3D single-channel image data) of the bounding box of the target category from the manual annotation. Randomly expand these six faces outward by 1 to 11 voxels, and crop them with the new bounding box to obtain the target size. Then, scale the image to 96×96×96 using trilinear interpolation. This preprocessing method increases the proportion of the target and makes the image size relatively uniform, suitable for uniform network parameters.

[0137] The six faces of the bounding box can be the six faces of a cube, namely the top, bottom, left, right, front, and back faces.

[0138] Step 3: Read the input image obtained in Step 2, perform data augmentation and further image preprocessing, that is, perform conventional data augmentation on the image to generate an augmented dataset, and add channel dimensions and standardize it, thereby increasing the diversity of the dataset and improving the generalization performance of the model, and keeping the data in the same distribution.

[0139] Here, the specific implementation of data augmentation is as follows: select one of the following options with uniform probability: flip along the x-axis, flip along the y-axis, flip along the z-axis, or do not flip.

[0140] The method for standardizing image data is Z-Score standardization, which transforms the enhanced image distribution into a standard normal distribution. The specific calculation method is as follows:

[0141]

[0142] Where x is the input image, x * The output image is defined as follows: μ is the mean of the pixel values ​​in the entire image, δ is the variance of the pixel values ​​in the entire image, and x is the result. * It follows a standard normal distribution.

[0143] Step 4: Divide the augmented dataset obtained in Step 3 into a training set and a test set in a ratio of 4:1. The training set is used to train the overall algorithm, and the test set is used to evaluate the segmentation results.

[0144] Step 5: Using the training set from Step 4 as input samples, read the multimodal images and concatenate them along the channel dimension. Then, input this concatenation into the multimodal coarse segmentation network 3D U-Net. This network includes a multimodal data aggregation module to aggregate multimodal data, three downsampling modules for feature encoding and reducing the number of computational parameters, a bridging module for calculating the deepest image features and bridging the upsampling and downsampling processes, three upsampling modules for feature decoding, and a final hierarchical prediction module for hierarchical prediction. Inputting the image into this network yields a coarse segmentation probability map.

[0145] Among them, the multimodal aggregation module, such as Figure 2 As shown, it includes an initialization submodule and a channel weighting submodule. The initialization submodule contains an IN operator and a convolutional layer with a kernel size of 1×1×1, a stride of 1, and padding of 0. The channel weighting submodule contains a pooling layer, two fully connected layers, and a pixel-wise multiplication operator. The pooling layer uses max pooling with a kernel size equal to the image size. The fully connected layer has 16 neurons. The pixel-wise multiplication operator assigns the weight values ​​from the fully connected layer to each channel of the feature.

[0146] Among them, the hierarchical prediction module, such as Figure 3 As shown, it contains three sub-prediction modules to output predicted labels at different levels. Each sub-prediction module includes four convolutional layers, with an activation operator after every two convolutional layers. Except for the last convolutional layer, the kernel size is 3×3×3, the convolution stride is 1, and the padding is 1. The first activation operator in each sub-prediction module is ReLU, and the second activation operator is Sigmoid.

[0147] Step 6: Based on the segmentation probability map and the ground truth of the samples, calculate the binary cross-entropy loss of lesions of different grades as the loss function of the network, and optimize it through the Adam optimizer with a learning rate of 3e-4. After optimization, output the coarse segmentation probability map of the training set and the test set.

[0148] In this scheme, the binary cross-entropy loss is calculated as follows:

[0149]

[0150]

[0151] Among them, L total Let C be the total loss, L be the number of lesion grades, P be the predicted probability, and L be the total loss. BCE Here is the formula for calculating binary cross-entropy loss, where Q is the label of a lesion of any grade. This is the prediction result for this sample.

[0152] Step 7: Feed back the segmentation probability map obtained in Step 6 to the medical experts, and the experts will provide the corresponding error correction information.

[0153] In this scheme, medical experts simulate the process using sample true values, calculate the error regions in the segmentation results using the true values, and randomly sample points in the error regions to simulate the expert's error correction points.

[0154] Step 8: Calculate the geodesic distance between each pixel in the multimodal original sample and the error correction point obtained in Step 7, and generate a distance metric map.

[0155] In this process, each simulated click error correction operation is considered a foreground point. The geodesic distance between each pixel in the image and the foreground point is calculated using the following formula:

[0156]

[0157] Where, x i and x j These are two pixels, P i,j Let be the set of all paths between two points, and p be a feasible path parameterized by s. It is a unit vector tangent to the direction of travel;

[0158] Step 9: Concatenate the distance metric map, segmentation results, and multimodal image samples along the channel dimension, and input the concatenated result into the multimodal interactive medical image segmentation network to obtain a new segmentation result;

[0159] The backbone network of the multimodal interactive medical image segmentation network is a three-dimensional PNet, and its network structure is as follows: Figure 4As shown, it consists of a multimodal aggregation module, five basic blocks, and a hierarchical prediction module. The input data is first processed by the multimodal aggregation module to adjust the weights of different modules and data with different distributions. Then, features are extracted from the data by the five basic blocks. The features output by the five basic blocks are then concatenated along the channel dimension. The concatenated result is input into the hierarchical prediction module. The results of the hierarchical prediction module are then concatenated along the channels to obtain the final segmentation probability map.

[0160] The multimodal aggregation module and hierarchical prediction module are the same as in step five, and will not be repeated here.

[0161] Of the five basic blocks, the first two basic blocks contain two dilated convolutions, and the last three basic blocks contain three dilated convolutions. The dilated convolutions within the same basic block have the same dilation rate, a kernel size of 3, a stride of 1, and padding equal to the dilation rate of the basic block plus 1. The dilation rates of each basic block are 0, 1, 2, 3, and 4 in sequence.

[0162] Step 10: Calculate the binary cross-entropy between the interactive segmentation result in Step 9 and the sample labels, and optimize it using the Adam optimizer with a learning rate of 3e-4. The calculation of the binary cross-entropy is the same as that in Step 6, and will not be repeated here.

[0163] Step 11: Input the test set samples from Step 4 into the multimodal coarse segmentation network to obtain the corresponding segmentation results. Feed back the segmentation results to human medical experts, who will then identify the corresponding error points. Calculate the geodesic distance between each pixel in the image and the error points. Then, stitch this distance metric map together with the segmentation results and the original sample channel dimensions. Input the stitched result into the multimodal interactive medical image segmentation network to obtain a new round of segmentation results. The feedback, interaction, and segmentation process iterates continuously until the maximum sequence length is exceeded or the medical experts deem the results to meet clinical standards.

[0164] The beneficial effects of this invention are as follows: the above technical solutions effectively integrate medical images of different modalities and solve the problem that binary interaction is difficult to adapt to multi-classification tasks in interactive segmentation. At the same time, this solution combines the two and effectively improves the accuracy of interactive segmentation methods by using multimodal medical image data, reduces the number of inferences required overall, makes expert interaction more efficient, and the improvement of results after interaction is more significant.

[0165] Example 3

[0166] Figure 6This is a structural block diagram of the multimodal interactive medical image segmentation device provided in Embodiment 3 of the present invention. The device can execute the multimodal interactive medical image segmentation method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the method.

[0167] like Figure 6 As shown, the device may include:

[0168] The preprocessing data determination unit 610 is used to read multimodal medical image input data and determine the cropping range according to the pre-labeled target category in order to crop the image and obtain preprocessing data of the target size.

[0169] The training set partitioning unit 620 is used to concatenate the preprocessed data along the channel dimension and divide the preprocessed data into a training set and a test set according to a preset ratio.

[0170] The segmentation probability map output unit 630 is used to input training set samples into a multimodal coarse segmentation network to obtain a segmentation probability map;

[0171] The final segmentation result determination unit 640 is used to obtain error correction suggestions for the segmentation probability map. If the segmentation probability map meets the preset standard, the segmentation probability map is output as the final segmentation result.

[0172] The multimodal distance metric map mapping unit 650 is used to determine the coordinates of the error points based on the error information in the error correction suggestions if the segmentation probability map does not meet the preset standard, and to map the error information into a multimodal distance metric map based on the coordinates of the error points.

[0173] The new segmentation probability map output unit 660 is used to stitch the multimodal distance metric map, multimodal medical image and segmentation probability map in the channel dimension, and send the stitching result into the multimodal interactive medical image segmentation network to obtain a new segmentation probability map;

[0174] The iterative statistics unit 670 is used to obtain error correction suggestions for the new segmentation probability map until the new segmentation probability map obtained by iteration exceeds the maximum sequence length or the segmentation probability map meets the standard.

[0175] The above-mentioned product can execute the multimodal interactive medical image segmentation method provided in the embodiments of this application, and has the corresponding functional modules and beneficial effects of the method.

[0176] Note that the above description is merely a preferred embodiment of the present invention and the technical principles employed. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions can be made without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments, and may include many other equivalent embodiments without departing from the concept of the present invention, the scope of which is determined by the scope of the appended claims.

Claims

1. A multimodal interactive medical image segmentation method, characterized in that, include: Read multimodal medical image input data and determine the cropping range according to the pre-labeled target category to crop the image and obtain preprocessed data of the target size; The preprocessed data is concatenated along the channel dimension, and then divided into a training set and a test set according to a preset ratio. The training set samples are input into a multimodal coarse segmentation network to obtain a segmentation probability map; Obtain error correction suggestions for the segmentation probability map; if the segmentation probability map meets the preset criteria, output the segmentation probability map as the final segmentation result. If the segmentation probability map does not meet the preset standard, then the coordinates of the error points are determined according to the error information in the error correction suggestions, and the error information is mapped into a multimodal distance metric map according to the coordinates of the error points; The multimodal distance metric map, multimodal medical image, and segmentation probability map are stitched together along the channel dimension, and the stitching result is fed into the multimodal interactive medical image segmentation network to obtain a new segmentation probability map. Obtain error correction suggestions for the new segmentation probability map until the new segmentation probability map obtained through iteration exceeds the maximum sequence length or the segmentation probability map meets the standard; The process includes determining the coordinates of error points based on the error information in the error correction suggestions, and mapping the error information into a multimodal distance metric map based on the coordinates of the error points, including: The system identifies mouse clicks in error areas, treating each click as a foreground point. It then calculates the geodesic distance between each pixel in the image and the foreground point, using the following formula: ； in, and Each is a two-pixel point. Let be the set of all paths between two points. For a Parameterized feasible paths It is a unit vector tangent to the path direction.

2. The method according to claim 1, characterized in that, After inputting the training set samples into a multimodal coarse segmentation network to obtain a segmentation probability map, the method further includes: By combining the segmentation probability map and the ground truth values ​​of the training set samples, the cross-entropy loss is calculated, and the cross-entropy loss is optimized using the gradient descent method.

3. The method according to claim 1, characterized in that, After feeding the stitched result into a multimodal interactive medical image segmentation network to obtain a new segmentation probability map, the method further includes: By combining the new segmentation probability map and the ground truth of the training set samples, the cross-entropy loss is calculated and optimized using gradient descent.

4. The method according to claim 1, characterized in that, The method further includes: The tumor is classified hierarchically according to different levels of lesions, and the changes in the tumor are modeled. The specific calculation formula is as follows: ； ； in, For the new label, The background class in the original tag. This refers to the edema category in the original label. The non-enhancing tumor category in the original label. This is an enhancement of the tumor category in the original label. For the transformed background class, For the transformed total tumor category, The transformed tumor core category, For the transformed enhanced tumor category, Let be the transformation function for each pixel. This represents the value of the pixel after matrix operations.

5. The method according to claim 1, characterized in that, After obtaining the preprocessed data for the target size, the method further includes: Data augmentation is performed on the preprocessed data of the target size to obtain an augmented dataset; wherein the data augmentation includes flipping along an arbitrary one-dimensional spatial coordinate axis.

6. The method according to claim 1, characterized in that, The backbone structure of the multimodal coarse segmentation network is a three-dimensional U-Net network, which includes a multimodal aggregation module and a hierarchical prediction module. The multimodal coarse segmentation network specifically includes: a multimodal aggregation module, three downsampling modules, one bridging module, three upsampling modules, and a hierarchical prediction module; The multimodal aggregation module includes an initialization submodule and a channel weighting submodule; The initialization submodule contains an IN operator and a convolutional layer with a kernel size of 1×1×1, a stride of 1, and padding of 0. The channel weighting submodule includes a pooling layer, two fully connected layers, and a pixel-wise multiplication operator; the pooling layer uses max pooling, and the pooling layer kernel size is the image size; the fully connected layer has 16 neurons; the pixel-wise multiplication operator is used to assign the weight values ​​after the fully connected layer to each channel of the feature. The downsampling module includes two convolutional layers and one pooling layer. The convolutional layer has a kernel size of 3×3×3, a stride of 1, and padding of 1. The pooling layer has a kernel size of 2×2×2 and a stride of 2. The bridging module includes two convolutional layers, each with a kernel size of 3×3×3, a stride of 1, and padding of 1. The upsampling module includes one deconvolution layer and two convolutional layers. The deconvolution operation has a kernel size of 3×3×3, a stride of 2, and padding of 1. The convolutional layers also have a kernel size of 3×3×3, a stride of 1, and padding of 1. The hierarchical prediction module includes three sub-prediction modules, each of which includes four convolutional layers. There is an activation operator after every two convolutional layers. Except for the last convolutional layer, the kernel size is 3×3×3, the convolution stride is 1, and the padding is 1. The first activation operator in each sub-prediction module is ReLU, and the second activation operator is Sigmoid.

7. The method according to claim 1, characterized in that, The multimodal interactive medical image segmentation network specifically includes: a multimodal aggregation module, five basic blocks, and a hierarchical prediction module; In the aforementioned basic blocks, the first two basic blocks contain two dilated convolutions, and the last three basic blocks contain three dilated convolutions. The dilated convolutions within the same basic block have the same dilation rate, a kernel size of 3, a stride of 1, and padding equal to the dilation rate of the basic block plus 1. The dilation rates of each basic block are 0, 1, 2, 3, and 4, respectively.

8. The method according to claim 3, characterized in that, Combining the new segmentation probability map and the ground truth values ​​of the training set samples, the cross-entropy loss is calculated, and optimized using gradient descent, including: The binary cross-entropy is calculated using the following formula: ; ； in, For the overall binary cross-entropy loss, The number of lesion grades. For real labels, To predict probabilities, Here is the formula for calculating binary cross-entropy loss. This is a label for lesions of any grade. This is the prediction result for this sample.

9. A multimodal interactive medical image segmentation device, characterized in that, include: The preprocessing data determination unit is used to read multimodal medical image input data and determine the cropping range according to the pre-labeled target category in order to crop the image and obtain preprocessed data of the target size; The training set partitioning unit is used to concatenate the preprocessed data along the channel dimension and divide the preprocessed data into a training set and a test set according to a preset ratio. The segmentation probability map output unit is used to input training set samples into the multimodal coarse segmentation network to obtain a segmentation probability map; The final segmentation result determination unit is used to obtain error correction suggestions for the segmentation probability map. If the segmentation probability map meets the preset standard, the segmentation probability map is output as the final segmentation result. A multimodal distance metric map mapping unit is used to determine the coordinates of the error points based on the error information in the error correction suggestions if the segmentation probability map does not meet the preset standard, and to map the error information into a multimodal distance metric map based on the coordinates of the error points. The new segmentation probability map output unit is used to stitch the multimodal distance metric map, multimodal medical image and segmentation probability map in the channel dimension, and send the stitching result into the multimodal interactive medical image segmentation network to obtain a new segmentation probability map; An iterative statistical unit is used to obtain error correction suggestions for the new segmentation probability map until the new segmentation probability map obtained through iteration exceeds the maximum sequence length or the segmentation probability map meets the standard. The process includes determining the coordinates of error points based on the error information in the error correction suggestions, and mapping the error information into a multimodal distance metric map based on the coordinates of the error points, including: The system identifies mouse clicks in error areas, treating each click as a foreground point. It then calculates the geodesic distance between each pixel in the image and the foreground point, using the following formula: ； in, and Each is a two-pixel point. Let be the set of all paths between two points. For a Parameterized feasible paths It is a unit vector tangent to the path direction.

Images