A medical image multi-scale refinement segmentation method and device

Abstract

The application discloses a kind of medical image multiscale fine segmentation method and device. Among them, the method comprises: reading original image input data, and using preset size to obtain preprocessing data by cutting;The preprocessing data is processed by linear interpolation method, to obtain target size and input to global rough segmentation network, to obtain rough segmentation probability graph;The rough segmentation probability graph is up-sampled to the original image size, and is spliced with the original image in channel dimension;The splicing result is input to intelligent area selection network, and the obtained area is selected, and the obtained area is input to local refinement network, to obtain refinement segmentation probability graph;The Dice gain before and after refinement is calculated as single-step reward value, and the refinement segmentation probability graph replaces the corresponding part in rough segmentation probability graph, to obtain reward value sequence, and obtain segmentation result based on reward value sequence.The scheme can achieve more efficient and accurate segmentation, complete the effect of medical image recognition task.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of machine learning and image processing, and particularly relates to a medical image multi-scale fine segmentation method and device. BACKGROUND

[0002] In recent years, with the rapid development of medical technology, the patient data in hospitals has increased dramatically, and a large amount of medical image data needs to be processed. The traditional manual recognition and analysis method, such as organ segmentation, tumor segmentation, target area demarcation and pathological analysis, has a high requirement on doctors, is too dependent on the state and experience of doctors, has the characteristics of high cost and low efficiency, and cannot meet the efficient data analysis demand, which brings great work pressure to doctors. In the face of massive medical images, how to quickly, efficiently and accurately identify the target in the medical images has become a research hotspot. SUMMARY

[0003] The present application provides a medical image multi-scale fine segmentation method and device, which can fully utilize the global and local information in the medical image, realize intelligent selection of fine regions through reinforcement learning, and achieve more efficient and accurate segmentation to complete the identification task of the medical image.

[0004] In a first aspect, the present application provides a medical image multi-scale fine segmentation method, which comprises:

[0005] reading original image input data, performing cropping preprocessing by using a preset size to obtain preprocessed data; dividing the preprocessed data into a training set and a test set according to a preset ratio;

[0006] processing the training set samples by using a linear interpolation method to obtain a target size, and inputting the training set samples of the target size into a global coarse segmentation network to obtain a coarse segmentation probability map;

[0007] upsampling the coarse segmentation probability map to the original image size, and splicing the coarse segmentation probability map with the original image in the channel dimension to obtain a splicing result;

[0008] inputting the splicing result into an intelligent region selection network to select a region, and inputting the region into a local refinement network to obtain a refined segmentation probability map;

[0009] calculating the Dice gain before and after refinement based on the refined segmentation probability map as a single-step reward value, replacing the region corresponding to the region in the coarse segmentation probability map with the refined segmentation probability map, splicing the refined segmentation probability map with the original image and inputting the refined segmentation probability map into the intelligent region selection network to calculate a reward value sequence, and obtaining a segmentation result based on the reward value sequence.

[0010] Further, after obtaining the coarse segmentation probability map, the method further comprises:

[0011] In combination with the coarse segmentation probability map and the true value of the training set sample, a cross-entropy loss is calculated, and the global coarse segmentation network is optimized by gradient descent method.

[0012] Further, in combination with the coarse segmentation probability map and the true value of the training set sample, a cross-entropy loss is calculated, and the global coarse segmentation network is optimized by gradient descent method, comprising:

[0013] Binary cross-entropy is used as the loss, the cross-entropy loss between different levels of lesions and the prediction results is calculated respectively, and the parameter optimization of the global coarse segmentation network is performed through back propagation.

[0014] Further, after the pre-processing of cropping with a preset size is performed, the method further comprises:

[0015] The pre-processed data is subjected to data enhancement processing to obtain an enhanced data set; wherein the data enhancement processing includes flipping processing along any one-dimensional spatial coordinate axis.

[0016] Further, comprising:

[0017] The backbone structure of the global coarse segmentation network is a U-Net network in three-dimensional form, specifically comprising: three down-sampling modules, a bridging module, three up-sampling modules, and an aggregation convolution module;

[0018] The down-sampling module comprises two convolution layers and a pooling layer, the convolution kernel size is 3x3x3, the convolution step is 1, and the padding is 1; the pooling layer kernel size is 2x2x2, and the pooling step is 2;

[0019] The bridging module comprises two convolution layers, the convolution kernel size of the convolution layer is 3x3x3, the convolution step is 1, and the padding is 1;

[0020] The up-sampling module comprises an inverse convolution layer and two convolution layers, the convolution kernel size of the inverse convolution layer is 3x3x3, the convolution step is 2, and the padding is 1; the convolution kernel size of the convolution layer is 3x3x3, the convolution step is 1, and the padding is 1;

[0021] The aggregation convolution module is set to a single convolution layer, the convolution kernel size is 1x1x1, the convolution step is 1, and the padding is 0;

[0022] The backbone structure of the local refinement network is a U-Net network in three-dimensional form, specifically comprising: three down-sampling modules, a bridging module, three up-sampling modules, and an aggregation convolution module;

[0023] The downsampling module includes two convolution layers and one pooling layer, the convolution kernel size is 3*3*3, the convolution step is 1, and the padding is 1; the pooling layer kernel size is 2*2*2, and the pooling step is 2;

[0024] The bridging module includes two convolution layers, the convolution kernel size of the convolution layer is 3*3*3, the convolution step is 1, and the padding is 1;

[0025] The upsampling module includes one deconvolution layer and two convolution layers, the convolution kernel size of the deconvolution layer is 3*3*3, the convolution step is 2, and the padding is 1, the convolution kernel size of the convolution layer is 3*3*3, the convolution step is 1, and the padding is 1;

[0026] The aggregation convolution module is set to a single convolution layer, the convolution kernel size is 1*1*1, the convolution step is 1, and the padding is 0.

[0027] Further, the backbone network of the intelligent region selection network is a double-branch convolutional neural network, specifically including:

[0028] four convolution modules, one feature extraction module, one Actor output end and one Critic output end;

[0029] The convolution module includes one convolution layer and one pooling layer, and one Elu operator, the convolution kernel size is 3*3*3, the convolution step is 1, and the padding is 1; the pooling kernel size is 2*2*2, and the pooling step is 2;

[0030] The feature extraction module includes two fully connected layers, and the number of neurons in the two fully connected layers is 12*10*8;

[0031] The Actor output end is composed of one fully connected layer, and the number of neurons is 6*5*4;

[0032] The Critic output end is composed of one fully connected layer, and the number of neurons is 1.

[0033] Further, the Dice gain before and after the refinement based on the refined segmentation probability map is calculated as a single-step reward value, including:

[0034] The reward value of the intelligent selection behavior is calculated by the following formula:

[0035] R t = Dice(P t ,L)- Dice(P t-1 ,L)+ I(P t ,L);

[0036]

[0037]

[0038] wherein, Dice(P t , L) is an index for measuring segmentation results, P t is a refined segmentation probability map at time step t, L is a real label of an image, R t is a reward value obtained at time step t, P is a coarse segmentation probability map generated by a global coarse segmentation network, x, y, z are region coordinates in a three-dimensional image, and I is a classification function of a real label.

[0039] Further, binary cross-entropy is used as a loss, and cross-entropy losses between different levels of lesions and prediction results are calculated respectively, including:

[0040] The formula for calculating the binary cross-entropy loss is as follows:

[0041]

[0042]

[0043] wherein, L total is a total loss, C is the number of lesion levels, L is a real label, P is a prediction probability, L BCE is a calculation formula of binary cross-entropy loss, Q is a label of an arbitrary level lesion, and P is a prediction result of the sample.

[0044] In a second aspect, the present application provides a multi-scale refined segmentation device for medical images, which comprises:

[0045] a preprocessing unit configured to read original image input data, perform cropping preprocessing by using a preset size, and obtain preprocessing data; and divide the preprocessing data into a training set and a test set according to a preset ratio;

[0046] a coarse segmentation unit configured to perform linear interpolation processing on the training set samples, obtain a target size, and input the training set samples of the target size into a global coarse segmentation network to obtain a coarse segmentation probability map;

[0047] a splicing unit configured to up-sample the coarse segmentation probability map to the size of the original image, and splice the coarse segmentation probability map with the original image in the channel dimension to obtain a splicing result;

[0048] a refined segmentation unit configured to input the splicing result into an intelligent region selection network, select a resulting region, and input the resulting region into a local refinement network to obtain a refined segmentation probability map;

[0049] The reward value sequence calculation unit calculates the Dice gain before and after the refinement based on the refined segmentation probability map as a single-step reward value, replaces the refined segmentation probability map in the part of the coarse segmentation probability map corresponding to the obtained region, and splices and inputs the original image to the intelligent region selection network to calculate the reward value sequence, and obtains the segmentation result based on the reward value sequence.

[0050] The technical solution provided in the present application realizes more efficient and accurate segmentation by preprocessing the sample, obtaining segmentation probability maps of different scales by using segmentation networks of different scales, and selecting local refined regions by the intelligent region selection network, and achieves the purpose of completing the recognition task of medical images. BRIEF DESCRIPTION OF DRAWINGS

[0051] Figure 1 is a flowchart of the medical image multi-scale refined segmentation method provided by the first embodiment of the present application;

[0052] Figure 2 is a schematic diagram of the global coarse segmentation network provided by the first embodiment of the present application;

[0053] Figure 3 is a schematic diagram of the intelligent region selection network provided by the first embodiment of the present application;

[0054] Figure 4 is a structural block diagram of the medical image multi-scale refined segmentation device provided by the third embodiment of the present application. DETAILED DESCRIPTION

[0055] The present application will be further described in detail below in conjunction with the drawings and embodiments. It can be understood that the specific embodiments described herein are only used to explain the present application, but not to limit the present application. In addition, it should be noted that, for the convenience of description, only the parts related to the present application are shown in the drawings, but not all the structures.

[0056] Before discussing the example embodiments in more detail, it is to be noted that some example embodiments are described as processes or methods depicted as flowcharts. Although the steps of the processes are depicted in a particular order, many of the steps can be performed concurrently, in parallel, or simultaneously. In addition, the order of the steps can be re-arranged. The processes can be terminated when their operations are completed, but the processes can also have additional steps not included in the figure. The processes can correspond to methods, functions, routines, subroutines, subprograms, etc.

[0057] Embodiment One

[0058] Figure 1This is a flowchart of a multi-scale fine segmentation method for medical images provided in Embodiment 1 of this application. This embodiment is applicable to scenarios involving the segmentation of lesion images. This method can be executed by the multi-scale fine segmentation device for medical images provided in this embodiment. The device can be implemented by software and / or hardware and can be integrated into an electronic device.

[0059] like Figure 1 As shown, the multi-scale refined segmentation method for medical images includes:

[0060] S110: Read the original image input data, perform cropping preprocessing using a preset size to obtain preprocessed data; divide the preprocessed data into training set and test set according to a preset ratio.

[0061] Specifically, the ratio of the training set to the validation set is 4:1. In order to enable medical images to better adapt to the unified network parameters and to remove unnecessary noise points and increase the ratio of the target to the background, this invention preprocesses all medical data when reading the input data.

[0062] When reading the input image, preprocessing is required to crop out areas where lesions may appear in the sample. Due to the unique nature of medical imaging, the edges of the sample often contain large amounts of black, image-free areas. These areas are directly cropped using bounding boxes, resulting in a cropped image size of 192×160×128. This preprocessing method increases the proportion of the target and makes the image size more uniform, suitable for consistent network parameters.

[0063] The six faces of the bounding box used during clipping can be the six faces of a cube, namely the top, bottom, left, right, front, and back faces.

[0064] In this embodiment, optionally, after preprocessing using a preset size to obtain preprocessed data, the method further includes:

[0065] The preprocessed data is subjected to data augmentation to obtain an augmented dataset; wherein the data augmentation includes flipping along an arbitrary one-dimensional spatial coordinate axis.

[0066] Specifically, the input image is read and subjected to data augmentation and further image preprocessing. This involves performing conventional data augmentation on the image to generate an augmented dataset, adding channel dimensions and standardizing it to increase the diversity of the dataset, improve the generalization performance of the model, and ensure that the data remains within a consistent distribution.

[0067] The specific implementation method for data augmentation here is as follows:

[0068] With uniform probability, choose one of the following spaces: flip along x-axis, flip along y-axis, flip along z-axis, no flip.

[0069] The data standardization of the image is Z-Score standardization, and the enhanced image distribution is converted into a standard normal distribution, and the specific calculation method is as follows:

[0070]

[0071] Wherein, x is the input image, x * is the output image, μ is the mean of the pixel value of the whole image, δ is the variance of the pixel value of the whole image, and the obtained x * obeys the standard normal distribution.

[0072] After obtaining the operation result, the preprocessed data needs to be divided into a training set and a test set according to a preset proportion. Specifically, the enhanced data set is divided into a training set and a test set according to a proportion of 4:1.

[0073] S120, the training set sample is processed by linear interpolation method to obtain the target size, and the training set sample of the target size is input into the global rough segmentation network to obtain the rough segmentation probability graph.

[0074] Specifically, the sample of the training set is scaled to 48x48x48 by the trilinear interpolation method, and then input into the global rough segmentation network to obtain the rough segmentation probability result, and upsampled to the original image size 192x160x128 by the trilinear interpolation method; Then, according to the segmentation probability graph and the true value of the sample, the cross-entropy loss is calculated as the loss function of the network.

[0075] In this scheme, optionally, the backbone structure of the global rough segmentation network is a U-Net network in three-dimensional form, specifically including: three down-sampling modules, a bridging module, three up-sampling modules and an aggregation convolution module;

[0076] The down-sampling module includes two convolution layers and a pooling layer, the convolution kernel size is 3x3x3, the convolution step is 1, and the padding is 1; the pooling layer kernel size is 2x2x2, and the pooling step is 2;

[0077] The bridging module includes two convolution layers, and the convolution kernel size of the convolution layer is 3x3x3, the convolution step is 1, and the padding is 1;

[0078] The up-sampling module includes an inverse convolution layer and two convolution layers, the convolution kernel size of the inverse convolution layer is 3x3x3, the convolution step is 2, and the padding is 1; the convolution kernel size of the convolution layer is 3x3x3, the convolution step is 1, and the padding is 1;

[0079] The aggregation convolution module is set as single-layer convolution, the convolution kernel size is 1x1x1, the convolution step is 1, and the padding is 0.

[0080] The backbone structure of the local refinement network is a three-dimensional U-Net network, specifically including three down-sampling modules, a bridging module, three up-sampling modules and an aggregation convolution module.

[0081] The down-sampling module includes two convolution layers and a pooling layer, the convolution kernel size is 3x3x3, the convolution step is 1, and the padding is 1; the pooling layer kernel size is 2x2x2, and the pooling step is 2.

[0082] The bridging module includes two convolution layers, and the convolution kernel size of the convolution layer is 3x3x3, the convolution step is 1, and the padding is 1.

[0083] The up-sampling module includes an inverse convolution layer and two convolution layers, the convolution kernel size of the inverse convolution layer is 3x3x3, the convolution step is 2, and the padding is 1; the convolution kernel size of the convolution layer is 3x3x3, the convolution step is 1, and the padding is 1.

[0084] The aggregation convolution module is set as single-layer convolution, the convolution kernel size is 1x1x1, the convolution step is 1, and the padding is 0.

[0085] Figure 2 The schematic diagram of the global coarse segmentation network provided by Embodiment One of the present application is shown in FIG. 1. Figure 2 As shown in FIG. 1, the three-dimensional U-Net network includes three down-sampling modules, a bridging module, three up-sampling modules and an aggregation convolution for output. The down-sampling module includes two convolution layers and a pooling layer, each convolution layer uses a 3x3x3 convolution kernel size, the step is 1, the padding is 1, the pooling step is 2, and the pooling size is 2x2x2.

[0086] The bridging module includes two convolution layers, each convolution layer uses a 3x3x3 convolution kernel, the step is 1, and the padding is 1. Different from the down-sampling module, the bridging module does not need to perform a pooling operation. The up-sampling module includes an inverse convolution operation and two convolution operations. The inverse convolution operation uses a 3x3x3 convolution kernel, the step is 2, and the padding is 1. The convolution operation uses a 3x3x3 convolution kernel, the step is 1, and the padding is 1. The aggregation convolution is single-layer convolution, the convolution kernel size is 1x1x1, the step is 1, and the padding is 0.

[0087] In this embodiment, the network used by the global coarse segmentation network includes three down-sampling modules for feature encoding and reducing the number of operation parameters, a bridge module for calculating the image features at the deepest level and bridging the up and down sampling processes, three up-sampling modules for decoding features, and an aggregation convolution for output. The image is input into the network to obtain an output probability map.

[0088] In this embodiment, specifically, the data is first input into the down-sampling module, and the results before the third pooling step are retained respectively. Then, the output of the last down-sampling module is input into the bridge module. After obtaining the output of the bridge module, it is input into the up-sampling module, but the input of the previous down-sampling module is sequentially spliced to the output after deconvolution. Finally, the result is obtained to perform aggregation convolution to obtain the required probability mapping P. In the labeled data, the cross-entropy loss is calculated using the probability map obtained in the foregoing and the true value, and is used as the loss function.

[0089] In this scheme, optionally, binary cross-entropy is used as the loss, and the cross-entropy loss between different levels of lesions and prediction results is calculated, including:

[0090] The formula for calculating the binary cross-entropy loss is as follows:

[0091]

[0092]

[0093] Wherein, L total is the total loss, C is the number of lesion levels, L is the true label, P is the prediction probability, L BCE is the calculation formula of the binary cross-entropy loss, Q is the annotation of the lesion at any level, is the prediction result of the sample.

[0094] S130, up-sampling the coarse segmentation probability map to the original image size, and splicing it with the original image in the channel dimension to obtain a splicing result.

[0095] In this embodiment, the probability map obtained in the foregoing with a size of 192x160x128 and its corresponding original image can be spliced in the channel dimension. Then, random cropping is performed on it, and the cropping window size is 32x32x32. Then, the cropping result is input into the local refinement network to obtain a refined segmentation probability map.

[0096] S140, inputting the splicing result into the intelligent region selection network, selecting the obtained region, and inputting the obtained region into the local refinement network to obtain a refined segmentation probability map.

[0097] The backbone structure of the local refinement network is the same as that of the global coarse segmentation network, and its structure is also consistent with that of the global coarse segmentation network, so it will not be explained in detail here. Similarly, the loss function used is the same as that used in the previous step, so it will not be explained in detail here either.

[0098] In this embodiment, optionally, the backbone network of the intelligent region selection network is a dual-branch convolutional neural network, specifically including:

[0099] Four convolutional modules, one feature extraction module, one Actor output, and one Critic output;

[0100] The convolution module includes a convolutional layer, a pooling layer, and an Elu operator. The convolutional kernel size is 3×3×3, the convolutional stride is 1, and the padding is 1. The pooling kernel size is 2×2×2, and the pooling stride is 2.

[0101] The feature extraction module includes two fully connected layers, each with 12×10×8 neurons.

[0102] The Actor output consists of a fully connected layer with 6×5×4 neurons.

[0103] The Critic output consists of a fully connected layer with one neuron.

[0104] Figure 3 This is a schematic diagram of the intelligent region selection network provided in Embodiment 1 of this application, as shown below. Figure 3 As shown, the backbone network of the intelligent region selection network is a dual-branch convolutional neural network, specifically including: four convolutional modules, one feature extraction module, one Actor output, and one Critic output. Each convolutional module comprises a convolutional layer, a pooling layer, and an Elu operator. The convolutional kernel size is 3×3×3, the convolutional stride is 1, and the padding is 1. The pooling kernel size is 2×2×2, and the pooling stride is 2. The feature extraction module includes two fully connected layers, each with 12×10×8 neurons. The Actor output consists of a fully connected layer with 6×5×4 neurons. The Critic output consists of a fully connected layer with 1 neuron.

[0105] S150, calculate the Dice gain before and after refinement based on the refined segmentation probability map as the single-step reward value, replace the part corresponding to the obtained region in the coarse segmentation probability map with the refined segmentation probability map, and calculate the reward value sequence by concatenating it with the original image and inputting it into the intelligent region selection network, and obtain the segmentation result based on the reward value sequence.

[0106] In this technical solution, optionally, the Dice gain before and after refinement is calculated as the single-step reward value based on the refined segmentation probability map, including:

[0107] The reward value for intelligent selection behavior is calculated using the following formula:

[0108] R t =Dice(P t ,L)-Dice(P t-1 ,L)+I(P t ,L);

[0109]

[0110]

[0111] Among them, Dice(P) t P(L) is an indicator for measuring the segmentation result. t Here is the refined segmentation probability map at time step t, where L is the true label of the image, and R is... t Let t be the reward value obtained at time step t, P be the coarse segmentation probability map generated by the global coarse segmentation network, x, y, z be the region coordinates in the 3D image, and I be the classification function of the true label.

[0112] The reward value is composed of the Dice coefficient gain before and after refining the region selected by the agent, plus a penalty factor. P is the probability map of the segmentation result generated by the coarse segmentation network, x, y, z are the region coordinates in the 3D image, and I is the penalty factor of the reward value function, which is calculated as the classification discriminant function of the real label.

[0113] In this scheme, the interaction between a single-step agent and the environment can be represented by three parts: region selection, local refinement, and feedback reward, with a maximum sequence length of 5.

[0114] The final complete inference process of this scheme is as follows: the samples of the test set are scaled down to the target size of the global coarse segmentation network and inferred to obtain coarse segmentation results. The segmentation results are then stitched together with the original image on the channel and input into the intelligent region selection network to obtain the selected regions. The selected regions are then input into the local refinement network. The process of selecting and refining regions is iterated until the maximum sequence length is exceeded.

[0115] In this scheme, to enhance the processing of image details, a global-local approach is adopted to simultaneously integrate global contextual information and local detail features of the image. Unlike global-local segmentation methods in natural images, to avoid using the sliding window method during inference, this scheme uses reinforcement learning to train an agent to select regions. This agent can effectively identify regions that the local thinning network can refine, rather than cropping the complete image by step size and then inputting it into the local thinning network. Therefore, the modified approach in this scheme is to select regions based on the output of the intelligent region selection network, then refine the image based on these regions, and replace the coarse segmentation result with the refined result. As the sequence length increases, the overall algorithm gradually pays more attention to local regions.

[0116] The method of this invention achieves a Dice of 89.91% for whole tumor (WT) segmentation on the Brain Tumor Segmentation Challenge 2019 (BraTS2019) dataset, a 4.72% improvement compared to the results achieved using only 3D U-Net on the same data. For tumor core (TC) segmentation, the Dice reaches 74.69%, a 12.18% improvement compared to the results achieved using only 3D U-Net on the same data. For enhancing tumor (ET) segmentation, the Dice reaches 68.24%, an 8.6% improvement compared to the results achieved using only 3D U-Net on the same data. BraTS2019 is a medical dataset for brain tumors, and Dice is a commonly used detection metric.

[0117] The technical solution provided in this application preprocesses the samples and uses a global coarse segmentation network, a local fine segmentation network, and an intelligent region selection network to build the overall model, thereby obtaining a systematic and refined medical image segmentation method, achieving the goal of quickly and accurately refining medical image segmentation.

[0118] Example 2

[0119] This embodiment is a preferred embodiment based on the above embodiments. The implementation example of the multi-scale refined segmentation method for medical images provided by this solution includes the following steps:

[0120] Step 1: When reading the input image, preprocessing is required, specifically cropping out areas in the sample that may contain lesions. Due to the unique nature of medical imaging, the edges of the sample often contain large amounts of black, image-free areas. These areas are directly cropped using bounding boxes, resulting in an image size of 192×160×128. This preprocessing method increases the proportion of the target and makes the image size more uniform, suitable for consistent network parameters.

[0121] The six faces of the bounding box used during clipping can be the six faces of a cube, namely the top, bottom, left, right, front, and back faces.

[0122] Step 2: Read the input image obtained in Step 1, perform data augmentation and further image preprocessing, that is, perform conventional data augmentation on the image to generate an augmented dataset, and add channel dimension and standardize it, thereby increasing the diversity of the dataset and improving the generalization performance of the model, and keeping the data in the same distribution.

[0123] The specific implementation method for data augmentation here is as follows:

[0124] Choose one of the following options with uniform probability: flip along the x-axis, flip along the y-axis, flip along the z-axis, or do not flip.

[0125] The method for standardizing image data is Z-Score standardization, which transforms the enhanced image distribution into a standard normal distribution. The specific calculation method is as follows:

[0126]

[0127] Where x is the input image, x * The output image is defined as follows: μ is the mean of the pixel values ​​in the entire image, δ is the variance of the pixel values ​​in the entire image, and x is the result. * It follows a standard normal distribution.

[0128] Step 3: Divide the augmented dataset obtained in Step 2 into a training set and a test set in a ratio of 4:1. The training set is used to train the overall algorithm, and the test set is used to evaluate the segmentation results.

[0129] Step 4: Further preprocess the training set obtained in Step 3 by scaling all samples to 48×48×48 using trilinear interpolation. Use this as the target input, read the image, and then input it into the global coarse segmentation network 3DU-Net. The network structure is as follows: Figure 2As shown, the network comprises three downsampling modules for feature encoding and reducing the number of computational parameters, a bridging module for calculating the deepest image features and bridging the upsampling and downsampling processes, three upsampling modules for feature decoding, and a final aggregation convolution for the output. Inputting an image into this network yields a coarse segmentation probability map.

[0130] Step 5: Upsample the coarse segmentation probability map to the original training set image size of 192×160×128 using trilinear interpolation; then, calculate the binary cross-entropy loss of different grades of lesions based on the segmentation probability map and the ground truth of the samples, using it as the loss function of the network, and optimize it using the Adam optimizer with a learning rate of 3e-4. After optimization, output the coarse segmentation probability maps of the training and test sets.

[0131] In this scheme, the binary cross-entropy loss is calculated as follows:

[0132]

[0133]

[0134] Among them, L total Let C be the total loss, L be the number of lesion grades, P be the predicted probability, and L be the total loss. BCE Here is the formula for calculating binary cross-entropy loss, where Q is the label of a lesion of any grade. This is the prediction result for this sample.

[0135] Step Six: To address the irreversible loss of information caused by downsampling preprocessing in medical image segmentation, this scheme introduces a local refinement network to receive global context information and perform refined segmentation based on local detail features. To this end, this scheme concatenates the segmentation probability map obtained in Step Five and the corresponding original sample image obtained in Step Three along the channel dimension, then randomly crops it with a cropping window size of 32×32×32. The cropped region is then input into the local refinement network to obtain the refined segmentation probability map. The network structure of this local refinement network is the same as the global coarse segmentation network in Step Four, and will not be repeated here.

[0136] Step 7: Calculate the binary cross-entropy between the refined region and the sample label in Step 6, and optimize it using the Adam optimizer with a learning rate of 3e-4. The calculation of the binary cross-entropy is the same as that in Step 5, and will not be repeated here.

[0137] Step 8: Concatenate the coarse segmentation probability map (192×160×128) from Step 5 with the samples from the dataset described in Step 3 along the channel dimension, and input it into the intelligent region selection network. The network structure is as follows: Figure 3 As shown, the network contains four convolutional modules for feature extraction, one feature extraction module for further feature extraction and dimensionality transformation, and finally two outputs for action prediction and value function prediction, respectively. After obtaining the selected region, the selected region is input into the local refinement network, and the Dice gain before and after refinement is calculated as the single-step reward value. The refined result replaces the coarse segmentation result, and then the region selection restarts until the sequence terminates, at which point the sequence reward is calculated.

[0138] In this embodiment, the specific formula for calculating the single-step reward value is as follows:

[0139] R t =Dice(P t ,L)-Dice(P t-1 ,L)+I(P t ,L);

[0140]

[0141]

[0142] Among them, Dice(P) t P(L) is an indicator for measuring the segmentation result. t P is the segmentation probability map at time step t. t-1 This is the segmentation probability map at time step t-1, i.e., the prior segmentation probability, where L is the true label of the image, and R... t denoted as the reward value obtained at time step t, which consists of the Dice coefficient gain before and after refinement of the region selected by the agent plus a penalty factor. P is the probability map of the segmentation result generated by the coarse segmentation network, x, y, z are the region coordinates in the 3D image, and I is the penalty factor of the reward value function, which is calculated as the classification discriminant function of the real label.

[0143] Step 9: Based on the sampling method in Step 8, the classic A3C algorithm is used to optimize the parameters of the intelligent region selection network. The Adam optimizer is used, with the learning rate set to 1e-4, the number of threads to 2, the training entropy coefficient to 0.01, the reward decay factor γ to 0.99, and the maximum sequence length to 5.

[0144] Step 10: Scale the test set samples from Step 3 to the target size of the global coarse segmentation network and infer the coarse segmentation result. Stitch the segmentation result with the original image on the channel and input it into the intelligent region selection network to obtain the selected region. Input the selected region into the local refinement network and iterate the process of selecting and refining the region until the maximum sequence length is exceeded.

[0145] The method of this invention achieves a Dice of 89.91% for whole tumor (WT) segmentation on the Brain Tumor Segmentation Challenge 2019 (BraTS2019) dataset, a 4.72% improvement compared to the results achieved using only 3D U-Net on the same data. For tumor core (TC) segmentation, the Dice reaches 74.69%, a 12.18% improvement compared to the results achieved using only 3D U-Net on the same data. For enhancing tumor (ET) segmentation, the Dice reaches 68.24%, an 8.6% improvement compared to the results achieved using only 3D U-Net on the same data. BraTS2019 is a medical dataset for brain tumors, and Dice is a commonly used detection metric.

[0146] The beneficial effects of this invention are as follows: This invention effectively alleviates the problem of missing local information in medical image segmentation through the above technical solutions. At the same time, the intelligent agent trained by reinforcement learning helps the local refinement algorithm to effectively improve its actual use efficiency, reduce the number of overall inferences, and make the selection of local refinement regions more accurate.

[0147] Example 3

[0148] Figure 4 This is a structural block diagram of a multi-scale fine segmentation device for medical images provided in Embodiment 3 of the present invention. This device can execute the multi-scale fine segmentation method for medical images provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the method.

[0149] like Figure 4 As shown, the device may include:

[0150] The preprocessing unit 410 is used to read the original image input data, perform cropping preprocessing using a preset size to obtain preprocessed data, and divide the preprocessed data into a training set and a test set according to a preset ratio.

[0151] The coarse segmentation unit 420 is used to process the training set samples using linear interpolation to obtain the target size, and input the training set samples of the target size into the global coarse segmentation network to obtain the coarse segmentation probability map.

[0152] The stitching unit 430 is used to upsample the coarse segmentation probability map to the original image size and stitch it with the original image in the channel dimension to obtain the stitching result.

[0153] The segmentation refinement unit 440 is used to input the stitching result into the intelligent region selection network, select the obtained region, and input the obtained region into the local refinement network to obtain a refined segmentation probability map.

[0154] The reward value sequence calculation unit 450 is used to calculate the Dice gain before and after refinement based on the refined segmentation probability map as a single-step reward value, replace the part of the coarse segmentation probability map corresponding to the obtained region with the refined segmentation probability map, and calculate the reward value sequence by concatenating it with the original image and inputting it into the intelligent region selection network, and obtain the segmentation result based on the reward value sequence.

[0155] The above-mentioned product can execute the multi-scale fine segmentation method for medical images provided in the embodiments of this application, and has the corresponding functional modules and beneficial effects of the method.

[0156] Note that the above description is merely a preferred embodiment of the present invention and the technical principles employed. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions can be made without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments, and may include many other equivalent embodiments without departing from the concept of the present invention, the scope of which is determined by the scope of the appended claims.

Claims

1. A multi-scale refined segmentation method for medical images, characterized in that, include: Read the original image input data, perform cropping preprocessing using preset dimensions, and obtain preprocessed data; The preprocessed data is divided into a training set and a test set according to a preset ratio; The training set samples are processed by linear interpolation to obtain the target size, and the training set samples of the target size are input into the global coarse segmentation network to obtain the coarse segmentation probability map. The coarse segmentation probability map is upsampled to the original image size and then stitched with the original image in the channel dimension to obtain the stitching result. The stitching result is input into the intelligent region selection network to select the obtained region, and the obtained region is input into the local refinement network to obtain a refined segmentation probability map; The Dice gain before and after refinement is calculated based on the refined segmentation probability map as the single-step reward value. The part corresponding to the obtained region in the coarse segmentation probability map is replaced by the refined segmentation probability map. The reward value sequence is calculated by concatenating it with the original image and inputting it into the intelligent region selection network. The segmentation result is obtained based on the reward value sequence. The backbone network of the intelligent region selection network is a two-branch convolutional neural network, specifically including: Four convolutional modules, one feature extraction module, one Actor output, and one Critic output; The convolution module includes a convolutional layer, a pooling layer, and an Elu operator. The convolutional kernel size is 3×3×3, the convolutional stride is 1, and the padding is 1. The pooling kernel size is 2×2×2, and the pooling stride is 2. The feature extraction module includes two fully connected layers, each with 12×10×8 neurons. The Actor output consists of a fully connected layer with 6×5×4 neurons. The Critic output consists of a fully connected layer with 1 neuron. The calculation of the Dice gain before and after refinement based on the refined segmentation probability map as the single-step reward value includes: The reward value for intelligent selection behavior is calculated using the following formula: R t =Dice(P t ,L)-Dice(P t-1 ,L)+I(P t ,L); Among them, Dice(P) t P(L) is an indicator for measuring the segmentation result. t Here is the refined segmentation probability map at time step t, where L is the true label of the image, and R is... t Let t be the reward value obtained at time step t, P be the coarse segmentation probability map generated by the global coarse segmentation network, x, y, z be the region coordinates in the 3D image, and I be the classification function of the true label.

2. The method according to claim 1, characterized in that, After obtaining the coarse segmentation probability map, the method further includes: By combining the coarse segmentation probability map and the ground truth of the training set samples, the cross-entropy loss is calculated, and the global coarse segmentation network is optimized using gradient descent.

3. The method according to claim 1, characterized in that, Combining the coarse segmentation probability map and the ground truth values ​​of the training set samples, the cross-entropy loss is calculated, and the global coarse segmentation network is optimized using gradient descent, including: Binary cross-entropy is used as the loss, and the cross-entropy loss between lesions of different grades and the prediction results is calculated. The parameters of the global coarse segmentation network are optimized by backpropagation.

4. The method according to claim 1, characterized in that, After preprocessing the data by cutting using preset dimensions, the method further includes: The preprocessed data is subjected to data augmentation to obtain an augmented dataset; wherein the data augmentation includes flipping along an arbitrary one-dimensional spatial coordinate axis.

5. The method according to claim 1, characterized in that, include: The backbone structure of the global coarse segmentation network is a three-dimensional U-Net network, specifically including: three downsampling modules, one bridging module, three upsampling modules, and an aggregation convolution module; The downsampling module includes two convolutional layers and one pooling layer. The convolutional kernel size is 3×3×3, the convolutional stride is 1, and the padding is 1. The pooling layer kernel size is 2×2×2, and the pooling stride is 2. The bridging module includes two convolutional layers, each with a kernel size of 3×3×3, a stride of 1, and padding of 1. The upsampling module includes one deconvolution layer and two convolutional layers. The deconvolution layer has a kernel size of 3×3×3, a stride of 2, and padding of 1. The convolutional layers have a kernel size of 3×3×3, a stride of 1, and padding of 1. The aggregated convolution module is set to a single-layer convolution with a kernel size of 1×1×1, a stride of 1, and padding of 0. The backbone structure of the local refinement network is a three-dimensional U-Net network, specifically including: three downsampling modules, one bridging module, three upsampling modules, and an aggregation convolution module; The downsampling module includes two convolutional layers and one pooling layer. The convolutional kernel size is 3×3×3, the convolutional stride is 1, and the padding is 1. The pooling layer kernel size is 2×2×2, and the pooling stride is 2. The bridging module includes two convolutional layers, each with a kernel size of 3×3×3, a stride of 1, and padding of 1. The upsampling module includes one deconvolution layer and two convolutional layers. The deconvolution layer has a kernel size of 3×3×3, a stride of 2, and padding of 1. The convolutional layers have a kernel size of 3×3×3, a stride of 1, and padding of 1. The aggregated convolution module is set to a single-layer convolution with a kernel size of 1×1×1, a stride of 1, and padding of 0.

6. The method according to claim 2, characterized in that, Using binary cross-entropy as the loss, the cross-entropy loss between lesions of different grades and the predicted results is calculated separately, including: The formula for calculating binary cross-entropy loss is as follows: Among them, L total Let C be the total loss, L be the number of lesion grades, P be the predicted probability, and L be the total loss. BCE Here is the formula for calculating binary cross-entropy loss, where Q is the label of a lesion of any grade. This is the prediction result for this sample.

7. A multi-scale fine segmentation device for medical images, characterized in that, include: The preprocessing unit is used to read the original image input data, perform cropping preprocessing using a preset size, and obtain preprocessed data. The preprocessed data is divided into a training set and a test set according to a preset ratio; The coarse segmentation unit is used to process the training set samples using linear interpolation to obtain the target size, and then input the training set samples of the target size into the global coarse segmentation network to obtain the coarse segmentation probability map. The stitching unit is used to upsample the coarse segmentation probability map to the original image size and stitch it with the original image in the channel dimension to obtain the stitching result; The segmentation refinement unit is used to input the stitching result into the intelligent region selection network, select the obtained region, and input the obtained region into the local refinement network to obtain a refined segmentation probability map; The reward value sequence calculation unit is used to calculate the Dice gain before and after refinement based on the refined segmentation probability map as a single-step reward value, replace the part of the coarse segmentation probability map corresponding to the obtained region with the refined segmentation probability map, and calculate the reward value sequence by concatenating it with the original image and inputting it into the intelligent region selection network, and obtain the segmentation result based on the reward value sequence. The backbone network of the intelligent region selection network is a two-branch convolutional neural network, specifically including: Four convolutional modules, one feature extraction module, one Actor output, and one Critic output; The convolution module includes a convolutional layer, a pooling layer, and an Elu operator. The convolutional kernel size is 3×3×3, the convolutional stride is 1, and the padding is 1. The pooling kernel size is 2×2×2, and the pooling stride is 2. The feature extraction module includes two fully connected layers, each with 12×10×8 neurons. The Actor output consists of a fully connected layer with 6×5×4 neurons. The Critic output consists of a fully connected layer with 1 neuron. The calculation of the Dice gain before and after refinement based on the refined segmentation probability map as the single-step reward value includes: The reward value for intelligent selection behavior is calculated using the following formula: R t =Dice(P t ,L)-Dice(P t-1 ,L)+I(P t ,L); Among them, Dice(P) t P(L) is an indicator for measuring the segmentation result. t Here is the refined segmentation probability map at time step t, where L is the true label of the image, and R is... t Let t be the reward value obtained at time step t, P be the coarse segmentation probability map generated by the global coarse segmentation network, x, y, z be the region coordinates in the 3D image, and I be the classification function of the true label.

Images