Three-dimensional medical image multi-organ reconstruction method based on context learning

Abstract

The invention discloses a three-dimensional medical image multi-organ reconstruction method based on context learning, and solves the problem of model domain migration in the prior art. The method comprises the following steps: acquiring a to-be-reasoned medical image and an image support set, preprocessing the to-be-reasoned medical image to obtain a preprocessed to-be-reasoned medical image, and preprocessing the image support set to obtain a preprocessed support set image; inputting the preprocessed to-be-reasoned medical image and the preprocessed support set image into the trained context multi-organ segmentation network for zero sample reasoning to obtain a plurality of processed images; performing post-processing according to the plurality of processed images to obtain a multi-organ three-dimensional reconstruction result; according to the method, multi-organ reconstruction of a downstream domain three-dimensional medical image can be completed without training by utilizing global context information provided by a support set.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a three-dimensional medical image multi-organ reconstruction method based on context learning. Background Art

[0002] At present, neural network architectures represented by U-Net are widely used in the field of medical image segmentation. U-Net networks and their derivative architectures have been successfully applied to a variety of medical image processing tasks, including fine segmentation of different anatomical regions, recognition of multiple structures and labels in a single region, image processing of different imaging modes (such as CT, MRI, etc.), and applications in a variety of imaging settings. These methods provide important support for clinical diagnosis and treatment by effectively segmenting complex medical image data. U-Net networks and their variants lack a mechanism for handling inter-domain differences. During training, they are accustomed to the feature distribution of the source domain. When applied to the target domain, the model cannot effectively align and adapt to the new feature distribution, resulting in reduced accuracy in segmentation tasks involving new anatomical structures, image modalities, or unknown labels.

[0003] Existing learning-based medical image segmentation methods have two important drawbacks. First, for most new segmentation tasks, models must be trained from scratch or existing models must be fine-tuned, which requires a lot of computing resources and deep learning expertise, which is not feasible for medical researchers and clinicians. Second, multi-task learning in the field of medical imaging may involve multiple modalities, different anatomical structures, etc., and the tasks are always determined by a pre-design. After training, the network can only solve the tasks defined during training. At the same time, current methods for few-shot learning in the field of medical imaging focus on limited data systems and handle specific tasks, including generalization to new classes in specific domains, such as abdominal CT or MRI. Summary of the invention

[0004] The present invention provides a method for multi-organ reconstruction of three-dimensional medical images based on contextual learning, which solves the problem in the prior art that the model needs to be trained or fine-tuned from scratch, and the existing few-sample learning methods are limited to specific data and tasks, resulting in the migration of the model domain to new categories or coping with new tasks. The network learning efficiency is improved, and the global contextual information additionally provided by the support set is used to enable the model to have stronger contextual reasoning capabilities, thereby achieving multi-organ reconstruction of three-dimensional medical images in the downstream domain without training.

[0005] The present invention provides a context-based learning-based 3D medical image multi-organ reconstruction method, the method comprising:

[0006] Acquire a medical image to be inferred and an image support set, and preprocess the medical image to be inferred to obtain a preprocessed medical image to be inferred, and preprocess the image support set to obtain a preprocessed support set image; wherein the preprocessed medical image to be inferred includes a plurality of medical image sub-blocks to be inferred, and the preprocessed support set image includes a plurality of image support set sub-blocks;

[0007] The preprocessed medical image to be inferred and the preprocessed support set image are input into the trained contextual multi-organ segmentation network for zero-sample inference to obtain a plurality of processed images; wherein the contextual multi-organ segmentation network comprises an encoder and a decoder connected in sequence; the encoder comprises: a plurality of feature multi-scale cross-fusion modules connected in sequence, and two identical down-sampling modules are connected between two feature multi-scale cross-fusion modules; the decoder comprises: a plurality of feature multi-scale cross-fusion modules connected in sequence, and two identical up-sampling modules are connected between two feature multi-scale cross-fusion modules; the encoder uses the down-sampling module and the multi-scale cross-fusion module to interactively process the preprocessed medical image to be inferred and the preprocessed support set image respectively, and extracts features from the preprocessed medical image to be inferred according to the interactive features to obtain a first extracted feature; the decoder uses the up-sampling module and the feature multi-scale cross-fusion module to restore the feature dimension of the first extracted feature to obtain a plurality of processed images;

[0008] Image post-processing is performed based on the multiple processed images to obtain a three-dimensional reconstruction result of multiple organs.

[0009] In a possible implementation, the obtaining of the medical image to be inferred and the image support set, and preprocessing the medical image to be inferred to obtain the preprocessed medical image to be inferred, includes:

[0010] Convert the data format of the medical image to be inferred, and determine whether the medical image to be inferred is complete. If so, resample the medical image to be inferred to obtain a resampled preprocessed medical image to be inferred; if not, process the medical image to be inferred;

[0011] Determine a minimum bounding box of the resampled preprocessed medical image to be inferred, and clip the resampled preprocessed medical image to be inferred according to the minimum bounding box to obtain a preprocessed sub-block image;

[0012] The preprocessed sub-block image is standardized to obtain a preprocessed medical image to be inferred.

[0013] In a possible implementation, the feature multi-scale cross fusion module is expressed as:

[0014] CrossBlock(u,V;θ z ,θ v )=(u′,V′);

[0015] Where u represents the training sub-block; V represents the image support set sub-block; θ z represents the first learnable parameter; θ v represents the second learnable parameter; u′ represents the training sub-block after interaction; V′ represents the support set sub-block after interaction.

[0016] In a possible implementation, the training process of the contextual multi-organ segmentation network includes:

[0017] Preprocessing the original medical image with labels to obtain a preprocessed image;

[0018] Dividing the preprocessed images into a data set to obtain a training set and a test set;

[0019] The contextual multi-organ segmentation network is trained according to the training set and the test set using a back-propagation training method to obtain a trained contextual multi-organ segmentation network.

[0020] In a possible implementation, preprocessing the original medical image with the label to obtain the preprocessed image includes:

[0021] Acquire an original medical image with a label, perform data format conversion and integrity check on the original medical image, and resample the original medical image with a label to obtain resampled data;

[0022] Clipping the resampled data, retaining a non-zero region of the clipped image, and obtaining a clipped image;

[0023] The cropped image is subjected to standard normalization to obtain a preprocessed image.

[0024] In a possible implementation, the training method using back propagation trains the contextual multi-organ segmentation network according to the training set and the test set to obtain a trained contextual multi-organ segmentation network, including:

[0025] Set the training batch, learning rate, and number of iterations;

[0026] Calculating a prediction result using the loss function of the contextual multi-organ segmentation network; calculating the loss between the prediction result and the label;

[0027] The contextual multi-organ segmentation network is optimized according to the loss to obtain a trained contextual multi-organ segmentation network.

[0028] In a possible implementation, the encoder uses the downsampling module and the multi-scale cross-fusion module to interactively process the preprocessed medical image to be inferred and the preprocessed support set image respectively, and extracts features of the preprocessed medical image to be inferred according to the interactive features to obtain first extracted features, including:

[0029] Splicing the preprocessed medical image to be inferred and the preprocessed support set image in the channel dimension to obtain a plurality of spliced ​​images;

[0030] Performing cross convolution processing on the multiple stitched images respectively to obtain initial interaction features corresponding to each stitched image;

[0031] Performing mean calculation on the initial interaction features corresponding to each spliced ​​image to obtain the interaction features corresponding to the preprocessed medical image to be inferred;

[0032] The pre-processed medical image to be inferred is resampled using the interactive feature and the down-sampling module to obtain a first extracted feature.

[0033] In a possible implementation, the loss function of the contextual multi-organ segmentation network is expressed as:

[0034]

[0035] Among them, L i (·) represents the target loss of the i-th layer; I represents the total number of layers of the decoder; f(X,S) represents; X represents the medical image to be inferred; S represents the image support set; Y represents the label.

[0036] In a possible implementation, the two identical downsampling modules respectively downsample the medical image to be inferred and the image support set through a multilayer perceptron; and the two identical upsampling modules respectively upsample the medical image to be inferred and the image support set through transposed convolution.

[0037] In a possible implementation, performing image post-processing according to the multiple processed images to obtain a multi-organ three-dimensional reconstruction result includes:

[0038] Assigning corresponding weights to the multiple processed images respectively, normalizing the weights of adjacent processed images, and performing weighted summation on all the multiple processed images to obtain an overall prediction result; wherein the weights obey a Gaussian distribution;

[0039] The overall prediction result is subjected to Gaussian blur processing, and is refined using a connected domain optimization algorithm to obtain a multi-organ three-dimensional reconstruction result.

[0040] One or more technical solutions provided in the present invention have at least the following technical effects or advantages:

[0041] (1) The present invention adopts a context-based learning-based 3D medical image multi-organ reconstruction method, combining the semantic guidance of the support set and the context information, thereby improving the generalization ability and prediction accuracy of the contextual multi-organ segmentation network;

[0042] (2) The present invention uses the global context information additionally provided by the image support set to enable the model to have stronger contextual reasoning capabilities, thereby realizing multi-organ reconstruction of downstream domain three-dimensional medical images without the need for training;

[0043] (3) The present invention utilizes a feature multi-scale cross-fusion module integrated into an encoder-decoder structure with residual connections to effectively integrate information across spatial scales. At the same time, a multi-strategy data enhancement scheme is adopted during the training process. The global context information provided by the support set is used to enable the model to have stronger contextual reasoning capabilities, thereby achieving high-precision multi-organ reconstruction of downstream domain three-dimensional medical images without training. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 A flowchart of the steps of a method for reconstructing multiple organs of three-dimensional medical images based on contextual learning provided by an embodiment of the present invention;

[0045] Figure 2 A schematic diagram of a feature multi-scale cross-fusion module provided in an embodiment of the present invention;

[0046] FIG3( a) is a slice visualization diagram of an abdominal multi-organ CT image provided by an embodiment of the present invention;

[0047] FIG3( b) is a slice diagram of the segmentation result of the abdominal multi-organ CT image provided by an embodiment of the present invention;

[0048] FIG3( c ) is a final 3D reconstruction of multiple abdominal organs provided by an embodiment of the present invention;

[0049] Figure 4 A schematic diagram of a contextual multi-organ segmentation network provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0050] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0051] The present invention provides a method for reconstructing multiple organs of three-dimensional medical images based on context learning, such as Figure 1 As shown, the method includes the following steps S101 to S103.

[0052] S101, obtaining a medical image to be inferred and an image support set, preprocessing the medical image to be inferred to obtain a preprocessed medical image to be inferred, and preprocessing the image support set to obtain a preprocessed support set image; wherein the preprocessed medical image to be inferred includes a plurality of medical image sub-blocks to be inferred, and the preprocessed support set image includes a plurality of image support set sub-blocks;

[0053] Specifically, in step S101, the medical image to be inferred is preprocessed to obtain a preprocessed medical image to be inferred, including: obtaining the medical image to be inferred and an image support set, and preprocessing the medical image to be inferred to obtain a preprocessed medical image to be inferred, including the following steps S1011 to S1013.

[0054] S1011, converting the data format of the medical image to be inferred, and determining whether the medical image to be inferred is complete. If so, resampling the medical image to be inferred to obtain a resampled preprocessed medical image to be inferred; if not, not processing the medical image to be inferred;

[0055] S1012, determining a minimum bounding box of the resampled preprocessed medical image to be inferred, and clipping the resampled preprocessed medical image to be inferred according to the minimum bounding box to obtain a preprocessed sub-block image;

[0056] S1013, standardizing the preprocessed sub-block image to obtain a preprocessed medical image to be inferred.

[0057] Exemplarily, the medical images to be inferred include common medical image formats such as DICOM (suffix .dcm), MHD (suffix .mhd and .raw) and NIFTY (suffix .nii or .nii.gz). To facilitate subsequent processing, all raw data are converted into a unified format, including a four-dimensional array and its corresponding compressed file.

[0058] The data integrity check determines whether the input is correct, including whether the geometric attributes of the data and labels, such as the origin, direction, size, and space, are consistent, whether there are missing values ​​in the label file, and whether the label file is continuous and free of abnormal values. If damage or abnormal values ​​are detected, the irreparable data will be directly removed, and the partially damaged data will be repaired by interpolation or filling; if there is no damaged data or abnormal values, the subsequent preprocessing operations will continue.

[0059] S102 inputs the preprocessed medical image to be inferred and the preprocessed support set image into the trained contextual multi-organ segmentation network for zero-sample inference to obtain multiple processed images; Figure 4 As shown, the contextual multi-organ segmentation network includes an encoder and a decoder connected sequentially.

[0060] The encoder includes: a plurality of sequentially connected feature multi-scale cross-fusion modules, such as Figure 2 The figure shows a feature multi-scale cross-fusion module, and two identical downsampling modules are connected between the two feature multi-scale cross-fusion modules.

[0061] The decoder comprises: a plurality of sequentially connected feature multi-scale cross-fusion modules, and two identical up-sampling modules are connected between two feature multi-scale cross-fusion modules.

[0062] The encoder uses a downsampling module and a multi-scale cross-fusion module to interactively process the preprocessed medical image to be inferred and the preprocessed support set image respectively, and extracts features of the preprocessed medical image to be inferred based on the interactive features to obtain the first extracted features.

[0063] The decoder uses an upsampling module and a feature multi-scale cross-fusion module to restore the feature dimension of the first extracted feature to obtain a plurality of processed images.

[0064] Specifically, in step S102, the training process of the contextual multi-organ segmentation network includes:

[0065] (1) Preprocessing the original medical image with labels to obtain a preprocessed image;

[0066] Here, the original medical image with labels is preprocessed to obtain the preprocessed image, including:

[0067] (1.1) Obtain the original medical images with labels, and perform data format conversion and integrity check on the original medical images;

[0068] (1.2) cropping the original medical image, retaining the non-zero area of ​​the cropped image, and obtaining a cropped image;

[0069] (1.3) Perform standard normalization on the cropped image to obtain the preprocessed image.

[0070] (2) Divide the preprocessed images into a data set to obtain a training set and a test set;

[0071] (3) The contextual multi-organ segmentation network is trained based on the training set and the test set using the back-propagation training method to obtain a trained contextual multi-organ segmentation network.

[0072] Here, the contextual multi-organ segmentation network is trained according to the training set and the test set using the back propagation training method to obtain the trained contextual multi-organ segmentation network, including:

[0073] (3.1) Set the training batch, learning rate and number of iterations;

[0074] (3.2) Calculate the prediction results using the loss function of the contextual multi-organ segmentation network; calculate the loss between the prediction results and the labels;

[0075] (3.3) The contextual multi-organ segmentation network is optimized according to the loss to obtain a trained contextual multi-organ segmentation network.

[0076] Exemplarily, step S1: preprocess the original medical images and labels, including CT, MRI, ultrasound, X-ray, etc. First, convert the data set format and check the data integrity; then resample to convert the image and label into a uniform spatial resolution; then crop the input image and label to retain the non-zero area; finally, normalize the image. The final multi-organ reconstruction accuracy and model generalization ability are improved by preprocessing the original image and label, which is specifically implemented as follows:

[0077] Step S11: converting the data set format and checking data integrity;

[0078] Common medical image formats include DICOM (suffix .dcm), MHD (suffix .mhd and .raw) and NIFTY (suffix .nii or .nii.gz). To facilitate subsequent processing, all raw data are converted into a unified format, including a four-dimensional array and its corresponding compressed file.

[0079] The data integrity check determines whether the input is correct, including whether the geometric attributes of the data and labels, such as the origin, direction, size, and space, are consistent, whether there are missing values ​​in the label file, and whether the label file is continuous and free of abnormal values. If damage or abnormal values ​​are detected, the irreparable data will be directly removed, and the partially damaged data will be repaired by interpolation or filling; if there is no damaged data or abnormal values, the subsequent preprocessing operations will continue.

[0080] Step S12: resampling data and labels;

[0081] First, the target resolution is determined by calculating the mean voxel spacing of all images in the dataset. Then, the image is resampled using cubic spline interpolation to retain the accuracy of details and edge information. Since the label is discrete category mask information, the nearest neighbor interpolation method is used for resampling to prevent the category label from being blurred. The resampling operation ensures the alignment and consistency of images and labels in the spatial dimension, while simplifying the complexity of processing data from different sources and resolutions.

[0082] Step S13: Then the input image and label are cropped to retain the non-zero area;

[0083] Find the smallest bounding box in the 3D medical image and the corresponding label. The range of this smallest bounding box is determined by the non-zero voxels of the image and label. These non-zero voxels represent the region of interest or the target organ. Then, the invalid area outside the bounding box area is cropped. At this time, the image and label are adjusted to a unified initial sub-block. The cropped image and label have no effect on the final segmentation result, but reducing the image and label size can significantly improve the computational efficiency of some data sets.

[0084] Step S14: Standardize the image.

[0085] When processing three-dimensional voxel data, convolutional neural networks may ignore the size differences of voxels in the actual physical space. By resizing image data and labels from different sources or different resolutions, it is possible to ensure that the actual physical space represented by each voxel is of the same size, avoiding the inability of the model to accurately learn spatial features due to resolution differences. Subsequently, the grayscale values ​​of the images are normalized to make the grayscale value distribution of each input image consistent, while the labels keep their discrete values ​​unchanged, because the pixel values ​​of the labels represent different category information and cannot be normalized. Z-score-based normalization includes the following steps:

[0086] Step S141: Mark the area to be segmented as a foreground category, collect the intensity information of all foreground voxels, and calculate the mean μ and standard deviation σ of these voxels:

[0087]

[0088] Among them, x i represents the i-th voxel value of the foreground area, and N represents the total number of foreground pixels;

[0089] Step S142: Normalize the resampled image data according to the result of step S141:

[0090]

[0091] in, represents the normalized i-th pixel value, μ represents the mean of the foreground voxels, and σ represents the standard deviation of the foreground voxels. That is, the voxel intensity normalization of all image data is completed by subtracting the foreground voxel mean and dividing by the standard deviation.

[0092] Step S2: performing training division on the preprocessed medical image data;

[0093] Step S21: Divide the overall data into a training set and a test set in a ratio of 4:1. Set the training batch to 4, with the first quarter of each training batch as the training image, and the remaining three quarters of the image-label pair data as the support set. In the preprocessing stage, the images and labels are adjusted to a unified sub-block, and all training images are randomly cropped or padded based on the central area to generate training sub-blocks with a resolution of 96×96×96 to ensure that each training sub-block contains foreground content, thereby improving the fitting efficiency of network training; the image-label pair sub-block as the support set is scaled to 96×96×96 to ensure that the support set can retain global information.

[0094] Step S22: Randomly crop each image in the test set data to obtain a test sub-block, where the resolution of the obtained test sub-block is the same as the training sub-block, and the test sub-block does not need to contain label content. This randomness is used to ensure the versatility of the test data. The support set in the test set is scaled to the same size as the test sub-block to retain global features.

[0095] Specifically, in step S102, the encoder uses a downsampling module and a multi-scale cross-fusion module to interactively process the pre-processed medical image to be inferred and the pre-processed support set image, respectively, and extracts features of the pre-processed medical image to be inferred based on the interactive features to obtain a first extracted feature; the decoder uses an upsampling module and a feature multi-scale cross-fusion module to restore the feature dimension of the first extracted feature to obtain multiple processed images; image post-processing is performed based on the multiple processed images to obtain a three-dimensional reconstruction result of multiple organs.

[0096] Specifically, in step S102, the feature multi-scale cross fusion module is expressed as:

[0097] CrossBlock(u,V;θ z ,θ v )=(u′,V′);

[0098] Where u represents the training sub-block; V represents; θ z Indicates; θ v represents; u′ represents; V′ represents.

[0099] For example, the training sub-block u and the support set sub-block v are iAs input, they are concatenated in the channel dimension so that the subsequent convolution can consider the features of both at the same time, and then pass through the cross convolution block to generate an interactive feature representation:

[0100] z i =A(CrossConv(u,v i θ z ))fori=1,2,…,n;

[0101] Among them, A(x) represents a nonlinear activation function, and CrossConv represents a convolution operation with learnable parameters θ. z The cross convolutional layers are used to capture the feature interactions between the input image and the support set.

[0102] The mean of the interaction features is calculated for all generated interaction features z i Take the mean to generate the updated input image feature u′:

[0103]

[0104] This step enables the features of the training sub-block to be interactively balanced with the features of the support set sub-block, while eliminating the excessive influence of a specific support set on the features of the training sub-block, thereby improving the adaptability and robustness of the model to diverse support sets.

[0105] In the image support set, for each interactive support set feature z i Extract new support set features v through the convolution layer i ', this feature can capture more contextual information:

[0106] v i '=A(Conv(z i θ v ))fori=1,2,…,n;

[0107] Among them, Conv(z i θ v ) is a tuple with learnable parameters θ v The convolution operation is used to extract the feature representation after interaction.

[0108] The feature multi-scale cross-fusion module enables each support set and input image to interact with other support sets and input images through their average values, extracting information from features of different scales and fusing them:

[0109] CrossBlock(u,V;θ z ,θ v )=(u′,V′);

[0110] Specifically, in step S102, two identical downsampling modules downsample the medical image to be inferred and the image support set respectively through a multi-layer perceptron; and two identical upsampling modules upsample the medical image to be inferred and the image support set respectively through transposed convolution.

[0111] Exemplarily, as shown in FIG3 , the encoder network has six layers, each of which uses a multi-layer perceptron structure based on three-dimensional convolution to extract features. The input of the network includes training sub-blocks and support set sub-blocks, which are spliced ​​in pairs in the channel dimension and then input into the encoder for feature extraction. The first layer of the encoder consists only of a feature multi-scale cross-fusion module, and the remaining layers consist of a feature multi-scale cross-fusion module and spatial downsampling of the training sub-blocks and support set sub-block representations, thereby achieving efficient feature extraction and representation. Spatial downsampling reduces the spatial resolution of the feature map through 3D convolution operations, while enhancing the depth of the features by increasing the number of convolutional layers. In this way, the encoder is able to gradually extract higher-level features while retaining lower-level detail information, thereby achieving efficient feature extraction and representation.

[0112] The decoder network has five layers, and the CNN decoder in the 3D U-Net network is used as the basis of the decoder network. Each layer of the decoder first spatially upsamples the training sub-block and the support set sub-block through transposed convolution to gradually restore the spatial structure and details of the image; then, the feature map of the decoder is spliced ​​with the feature map of the same size in the encoder path in the channel dimension through jump connections. This jump connection can directly pass the high-level features extracted from the encoder to the decoder, ensuring that the spatial details are preserved and enhanced at high resolution; then the spliced ​​feature map will pass through the feature multi-scale cross fusion module to effectively fuse information of different scales, so that each feature can fully express the global and local feature details. At the end of the decoder, a single 1×1 convolution is used to map the fused feature map to the required number of output categories to form the final segmentation prediction map. The whole process ensures the effective fusion of high-resolution features and multi-scale information through step-by-step upsampling, jump connections and feature multi-scale cross fusion modules, thereby improving the accuracy and reliability of the segmentation results.

[0113] The encoder and decoder are cascaded in sequence, and the output of the encoder is fused with the input of the decoder by skip connection at each layer, and the high-resolution features are retained through residual connection. By introducing deep supervision, feature interaction between the encoder and decoder is established, thus forming a medical image multi-organ reconstruction network with contextual understanding ability. Deep supervision refers to applying loss functions to multiple intermediate layers of the network, which not only supervises the final output, but also directly optimizes the feature representation of the intermediate layer, thereby improving gradient flow, improving the convergence speed and segmentation accuracy of the network. The loss function of the contextual multi-organ segmentation network is expressed as:

[0114]

[0115] Among them, L i (·) represents the target loss of the i-th layer; I represents the total number of layers of the decoder; f(X,S) represents; X represents the medical image to be inferred; S represents the image support set; Y represents the label.

[0116] The output of layer 0 has the highest resolution, and the output of layer I has the lowest resolution.

[0117] S103, performing image post-processing according to the plurality of processed images to obtain a multi-organ three-dimensional reconstruction result.

[0118] Specifically, in step S103, image post-processing is performed based on the multiple processed images to obtain a multi-organ three-dimensional reconstruction result, including:

[0119] (1) assigning corresponding weights to the multiple processed images respectively, normalizing the weights of adjacent processed images, and performing weighted summation on all the multiple processed images to obtain an overall prediction result; wherein the weights obey a Gaussian distribution;

[0120] (2) The overall prediction results are processed by Gaussian blur and refined using the connected domain optimization algorithm to obtain the three-dimensional reconstruction results of multiple organs.

[0121] Exemplarily, the prediction of 3D medical image multi-organ reconstruction in downstream tasks is realized. The images and support sets to be inferred are input into the trained context-based learning-based 3D medical image multi-organ reconstruction network to perform zero-sample inference using the global context information obtained in the network. In this process, accurate prediction of data in different domains can be achieved without training, thereby generating high-quality image sub-blocks.

[0122] The realization of zero-shot reasoning relies on the extensive contextual understanding ability acquired by the network during the training phase, especially in multi-organ reconstruction tasks. By combining multi-level contextual information and support set hints, the network can directly reason without updating weights during the reasoning process. Since the network has accumulated rich contextual information and generalization capabilities during the training phase, it can directly apply these experiences to reasoning tasks in different domains to ensure cross-domain versatility. It is precisely because the network can use the extensive semantic and structural information obtained in the previous training process that it can efficiently reason about new image domains and generate accurate segmentation results, thereby greatly improving the efficiency and adaptability of the reasoning process.

[0123] Assign a weight that conforms to the Gaussian distribution to each predicted 3D image sub-block, normalize the weights of adjacent predicted sub-blocks, and perform weighted summation on all predicted sub-blocks to obtain the overall prediction result;

[0124] The predicted segmentation results are Gaussian blurred to smooth the edges, and the connected domain optimization algorithm is applied to further refine the results, ultimately generating a complete and accurate three-dimensional multi-organ reconstructed image.

[0125] Specifically, back propagation training is performed on the contextual multi-organ segmentation network that has completed initial training to obtain a trained contextual multi-organ segmentation network.

[0126] Here, the contextual multi-organ segmentation network that has completed the initial training is back-propagated to obtain a trained context-based learning three-dimensional medical image multi-organ segmentation network.

[0127] Step 1: First, set the training batch size to 4, use IN as the normalization operation in the forward propagation, use the AdamW optimizer, and set the initial learning rate to 0.001. The training process is set to 500 iterations, and the activation function is Leaky ReLU. During the training process, the number of support sets for each sample is fixed to 4; different numbers of candidates are considered during inference.

[0128] Step 2: After the network weights are forward propagated, the loss between the prediction result and the label is calculated through the loss function;

[0129] Step 3: Repeat step 2 until the loss reaches convergence, then stop training and select the node weight with the highest verification score during training as the final network weight.

[0130] The effect of the present invention can be further illustrated by the following simulation results:

[0131] 1. Simulation conditions

[0132] The multi-organ reconstruction network designed during the simulation process is based on Python 3.10 and Pytorch 2.0.0 framework;

[0133] The machine used to train the reconstruction network during the simulation is equipped with 2 GeForce RTX V100 GPUs;

[0134] The dataset used in the simulation is an abdominal multi-organ CT image and label dataset obtained from the public BTCV dataset.

[0135] 2. Simulation Content

[0136] Under the above simulation conditions, the method of the present invention is used to perform three-dimensional intelligent reconstruction on the obtained abdominal multi-organ CT images and label data sets, wherein:

[0137] FIG3( a ) is a slice visualization of an abdominal multi-organ CT image;

[0138] FIG3( b ) is a slice diagram of the segmentation result of the abdominal multi-organ CT image of the present invention;

[0139] Figure 3(c) is the final 3D reconstruction of the abdominal multi-organs, which includes 13 regions, including the spleen, right kidney, left kidney, gallbladder, esophagus, liver, stomach, aorta, inferior vena cava, portal vein and splenic vein, pancreas, right adrenal gland, and left adrenal gland;

[0140] It can be observed from FIG3(a) that the resolution between the organs in the abdominal region is low and the boundaries are blurred. FIG3(b) shows the segmentation effect of the present invention, which can automatically and accurately segment multiple organs in the abdomen, effectively avoiding the interference of a large amount of blurred background, and the edges between the organs are also clearer. FIG3(c) further shows the final reconstructed three-dimensional structure of multiple organs in the abdomen, clearly showing the relative position and shape of each organ.

[0141] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. All or part of the present invention can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld devices or portable devices, tablet devices, mobile communication terminals, multi-processor systems, microprocessor-based systems, programmable electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc.

[0142] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some or all of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the present invention.

Claims

1. A method for multi-organ reconstruction of three-dimensional medical images based on contextual learning, characterized in that: include: Acquire a medical image to be inferred and an image support set, and preprocess the medical image to be inferred to obtain a preprocessed medical image to be inferred, and preprocess the image support set to obtain a preprocessed support set image; wherein the preprocessed medical image to be inferred includes a plurality of medical image sub-blocks to be inferred, and the preprocessed support set image includes a plurality of image support set sub-blocks; The preprocessed medical image to be inferred and the preprocessed support set image are input into the trained contextual multi-organ segmentation network for zero-sample inference to obtain a plurality of processed images; wherein the contextual multi-organ segmentation network comprises an encoder and a decoder connected in sequence; the encoder comprises: a plurality of feature multi-scale cross-fusion modules connected in sequence, and two identical down-sampling modules are connected between two feature multi-scale cross-fusion modules; the decoder comprises: a plurality of feature multi-scale cross-fusion modules connected in sequence, and two identical up-sampling modules are connected between two feature multi-scale cross-fusion modules; the encoder uses the down-sampling module and the multi-scale cross-fusion module to interactively process the preprocessed medical image to be inferred and the preprocessed support set image respectively, and extracts features from the preprocessed medical image to be inferred according to the interactive features to obtain a first extracted feature; the decoder uses the up-sampling module and the feature multi-scale cross-fusion module to restore the feature dimension of the first extracted feature to obtain a plurality of processed images; Image post-processing is performed based on the multiple processed images to obtain a three-dimensional reconstruction result of multiple organs.

2. The context-based learning-based 3D medical image multi-organ reconstruction method according to claim 1, characterized in that: The obtaining of the medical image to be inferred and the image support set, and preprocessing the medical image to be inferred to obtain the preprocessed medical image to be inferred, includes: Convert the data format of the medical image to be inferred, and determine whether the medical image to be inferred is complete. If so, resample the medical image to be inferred to obtain a resampled preprocessed medical image to be inferred; if not, process the medical image to be inferred; Determine a minimum bounding box of the resampled preprocessed medical image to be inferred, and clip the resampled preprocessed medical image to be inferred according to the minimum bounding box to obtain a preprocessed sub-block image; The preprocessed sub-block image is standardized to obtain a preprocessed medical image to be inferred.

3. The context-based learning-based 3D medical image multi-organ reconstruction method according to claim 1, characterized in that: The feature multi-scale cross fusion module is expressed as: CrossBlock(u,V;θ z ,θ v )=(u′,V′); Where u represents the training sub-block; V represents the image support set sub-block; θ z represents the first learnable parameter; θ v represents the second learnable parameter; u′ represents the training sub-block after interaction; V′ represents the support set sub-block after interaction.

4. The method for 3D medical image multi-organ reconstruction based on context learning according to claim 1, characterized in that: The training process of the contextual multi-organ segmentation network includes: Preprocessing the original medical image with labels to obtain a preprocessed image; Dividing the preprocessed images into a data set to obtain a training set and a test set; The contextual multi-organ segmentation network is trained according to the training set and the test set using a back-propagation training method to obtain a trained contextual multi-organ segmentation network.

5. The context-based learning-based 3D medical image multi-organ reconstruction method according to claim 4, characterized in that: The preprocessing of the original medical image with the label to obtain the preprocessed image includes: Acquire an original medical image with a label, perform data format conversion and integrity check on the original medical image, and resample the original medical image with a label to obtain resampled data; Clipping the resampled data, retaining a non-zero region of the clipped image, and obtaining a clipped image; The cropped image is subjected to standard normalization to obtain a preprocessed image.

6. The method for 3D medical image multi-organ reconstruction based on context learning according to claim 4, characterized in that: The training method using back propagation trains the contextual multi-organ segmentation network according to the training set and the test set to obtain a trained contextual multi-organ segmentation network, including: Set the training batch, learning rate, and number of iterations; Calculating a prediction result using the loss function of the contextual multi-organ segmentation network; calculating the loss between the prediction result and the label; The contextual multi-organ segmentation network is optimized according to the loss to obtain a trained contextual multi-organ segmentation network.

7. The context-based learning-based 3D medical image multi-organ reconstruction method according to claim 1, characterized in that: The encoder uses the downsampling module and the multi-scale cross-fusion module to interactively process the preprocessed medical image to be inferred and the preprocessed support set image respectively, and extracts features from the preprocessed medical image to be inferred according to the interactive features to obtain first extracted features, including: Splicing the preprocessed medical image to be inferred and the preprocessed support set image in the channel dimension to obtain a plurality of spliced ​​images; Performing cross convolution processing on the multiple stitched images respectively to obtain initial interaction features corresponding to each stitched image; Performing mean calculation on the initial interaction features corresponding to each spliced ​​image to obtain the interaction features corresponding to the preprocessed medical image to be inferred; The pre-processed medical image to be inferred is resampled using the interactive feature and the down-sampling module to obtain a first extracted feature.

8. The context-based learning-based 3D medical image multi-organ reconstruction method according to claim 1, characterized in that: The loss function of the contextual multi-organ segmentation network is expressed as: Among them, L i (·) represents the target loss of the i-th layer; I represents the total number of layers of the decoder; f(X,S) represents; X represents the medical image to be inferred; S represents the image support set; Y represents the label.

9. The context-based learning-based 3D medical image multi-organ reconstruction method according to claim 1, characterized in that: The two identical downsampling modules downsample the medical image to be inferred and the image support set respectively through a multi-layer perceptron; the two identical upsampling modules upsample the medical image to be inferred and the image support set respectively through transposed convolution.

10. The context-based learning-based 3D medical image multi-organ reconstruction method according to claim 1, characterized in that: The performing image post-processing according to the plurality of processed images to obtain a multi-organ three-dimensional reconstruction result includes: Assigning corresponding weights to the multiple processed images respectively, normalizing the weights of adjacent processed images, and performing weighted summation on all the multiple processed images to obtain an overall prediction result; wherein the weights obey a Gaussian distribution; The overall prediction result is subjected to Gaussian blur processing, and is refined using a connected domain optimization algorithm to obtain a multi-organ three-dimensional reconstruction result.

Images