A 3D medical image multi-organ reconstruction method based on contextual learning

Abstract

The present invention discloses a method for reconstructing multiple organs of three-dimensional medical images based on context learning, which solves the problem of model domain migration in the prior art. The method comprises: obtaining a medical image to be inferred and an image support set, preprocessing the medical image to be inferred to obtain a preprocessed medical image to be inferred, and preprocessing the image support set to obtain a preprocessed support set image; inputting the preprocessed medical image to be inferred and the preprocessed support set image into a trained context multi-organ segmentation network for zero-sample inference to obtain a plurality of processed images; performing post-processing based on the plurality of processed images to obtain a multi-organ three-dimensional reconstruction result; the method utilizes the global context information provided by the support set to realize the multi-organ reconstruction of the downstream domain three-dimensional medical image without training.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a three-dimensional medical image multi-organ reconstruction method based on context learning. Background Art

[0002] Currently, neural network architectures, represented by U-Net, are widely used in the field of medical image segmentation. U-Net networks and their derivative architectures have been successfully applied to a variety of medical image processing tasks, including fine segmentation of different anatomical regions, recognition of multiple structures and labels within a single region, image processing of different imaging modalities (such as CT, MRI, etc.), and application in a variety of imaging settings. These methods provide important support for clinical diagnosis and treatment by effectively segmenting complex medical imaging data. U-Net networks and their variants lack mechanisms for handling inter-domain differences. During training, they become accustomed to the feature distribution of the source domain. When applied to the target domain, the model cannot effectively align and adapt to the new feature distribution, resulting in reduced accuracy in segmentation tasks involving new anatomical structures, image modalities, or unknown labels.

[0003] Existing learning-based medical image segmentation methods have two important shortcomings. First, for most new segmentation tasks, models must be trained from scratch or existing models must be fine-tuned, which requires extensive computational resources and deep learning expertise and is not feasible for medical researchers and clinicians. Second, multi-task learning in the field of medical imaging may involve multiple modalities, different anatomical structures, etc., and the tasks are always pre-designed and determined. After training, the network can only solve the tasks defined during training. At the same time, current few-shot learning methods in the field of medical imaging focus on limited data systems and handle specific tasks, including generalization to new classes in a specific domain, such as abdominal CT or MRI. Summary of the Invention

[0004] The present invention provides a method for multi-organ reconstruction of three-dimensional medical images based on contextual learning, which solves the problems in the prior art that the model needs to be trained or fine-tuned from scratch, and the existing few-sample learning methods are limited to specific data and tasks, resulting in the model domain migrating to new categories or coping with new tasks. It improves the network learning efficiency and utilizes the global contextual information additionally provided by the support set to give the model stronger contextual reasoning capabilities, thereby realizing multi-organ reconstruction of three-dimensional medical images in the downstream domain without training.

[0005] The present invention provides a context-based learning-based 3D medical image multi-organ reconstruction method, the method comprising:

[0006] Acquiring a medical image to be inferred and an image support set, preprocessing the medical image to be inferred to obtain a preprocessed medical image to be inferred, and preprocessing the image support set to obtain a preprocessed support set image; wherein the preprocessed medical image to be inferred includes a plurality of medical image sub-blocks to be inferred, and the preprocessed support set image includes a plurality of image support set sub-blocks;

[0007] The preprocessed medical image to be inferred and the preprocessed support set image are input into the trained contextual multi-organ segmentation network for zero-sample inference to obtain a plurality of processed images; wherein the contextual multi-organ segmentation network includes an encoder and a decoder connected in sequence; the encoder includes: a plurality of feature multi-scale cross-fusion modules connected in sequence, and two identical downsampling modules are connected between two feature multi-scale cross-fusion modules; the decoder includes: a plurality of feature multi-scale cross-fusion modules connected in sequence, and two identical upsampling modules are connected between two feature multi-scale cross-fusion modules; the encoder uses the downsampling module and the multi-scale cross-fusion module to interactively process the preprocessed medical image to be inferred and the preprocessed support set image respectively, and extracts features from the preprocessed medical image to be inferred based on the interactive features to obtain a first extracted feature; the decoder uses the upsampling module and the feature multi-scale cross-fusion module to restore the feature dimension of the first extracted feature to obtain a plurality of processed images;

[0008] The feature multi-scale cross fusion module is expressed as:

[0009] ;

[0010] in, represents the training sub-block; Represents the image support set sub-block; represents the first learnable parameter; represents the second learnable parameter; represents the training sub-block after interaction; represents the support set block after interaction;

[0011] The loss function of the contextual multi-organ segmentation network is expressed as:

[0012] ;

[0013] in, Indicates the The target loss of the layer; Indicates the total number of decoder layers; represents the medical image to be inferred; represents the image support set; Indicates a label;

[0014] The encoder uses the downsampling module and the multi-scale cross fusion module to interactively process the preprocessed medical image to be inferred and the preprocessed support set image respectively, and extracts features from the preprocessed medical image to be inferred based on the interactive features to obtain first extracted features, including:

[0015] Splicing the preprocessed medical image to be inferred and the preprocessed support set image in the channel dimension to obtain a plurality of spliced ​​images;

[0016] Performing cross convolution processing on the multiple stitched images respectively to obtain initial interaction features corresponding to each stitched image;

[0017] Performing mean calculation on the initial interaction features corresponding to each spliced ​​image to obtain the interaction features corresponding to the preprocessed medical image to be inferred;

[0018] Resampling the pre-processed medical image to be inferred using the interactive feature and the downsampling module to obtain a first extracted feature;

[0019] Image post-processing is performed based on the multiple processed images to obtain a three-dimensional reconstruction result of multiple organs.

[0020] In a possible implementation, obtaining the medical image to be inferred and the image support set, and preprocessing the medical image to be inferred to obtain the preprocessed medical image to be inferred, includes:

[0021] Converting the data format of the medical image to be inferred and determining whether the medical image to be inferred is complete; if so, resampling the medical image to be inferred to obtain a resampled preprocessed medical image to be inferred; if not, processing the medical image to be inferred;

[0022] Determining a minimum bounding box of the resampled preprocessed medical image to be inferred, and cropping the resampled preprocessed medical image to be inferred according to the minimum bounding box to obtain a preprocessed sub-block image;

[0023] The preprocessed sub-block image is standardized to obtain a preprocessed medical image to be inferred.

[0024] In one possible implementation, the training process of the contextual multi-organ segmentation network includes:

[0025] Preprocessing the original medical image with labels to obtain a preprocessed image;

[0026] Dividing the preprocessed images into a training set and a test set;

[0027] The contextual multi-organ segmentation network is trained according to the training set and the test set using a back-propagation training method to obtain a trained contextual multi-organ segmentation network.

[0028] In a possible implementation, preprocessing the original medical image with the label to obtain the preprocessed image includes:

[0029] Obtaining an original medical image with a label, performing data format conversion and integrity check on the original medical image, and resampling the original medical image with a label to obtain resampled data;

[0030] Clipping the resampled data, retaining a non-zero region of the clipped image, and obtaining a clipped image;

[0031] Performing standard normalization on the cropped image to obtain a preprocessed image.

[0032] In a possible implementation, the training method using back propagation trains the contextual multi-organ segmentation network according to the training set and the test set to obtain a trained contextual multi-organ segmentation network, including:

[0033] Set the training batch, learning rate, and number of iterations;

[0034] Calculating a prediction result using the loss function of the contextual multi-organ segmentation network; calculating the loss between the prediction result and the label;

[0035] The contextual multi-organ segmentation network is optimized according to the loss to obtain a trained contextual multi-organ segmentation network.

[0036] In one possible implementation, the two identical downsampling modules downsample the medical image to be inferred and the image support set respectively through a multi-layer perceptron; and the two identical upsampling modules upsample the medical image to be inferred and the image support set respectively through transposed convolution.

[0037] In a possible implementation, performing image post-processing based on the multiple processed images to obtain a multi-organ three-dimensional reconstruction result includes:

[0038] Assigning corresponding weights to the multiple processed images respectively, normalizing the weights of adjacent processed images, and performing weighted summation on all the multiple processed images to obtain an overall prediction result; wherein the weights obey a Gaussian distribution;

[0039] The overall prediction result is subjected to Gaussian blur processing and refined using a connected domain optimization algorithm to obtain a multi-organ three-dimensional reconstruction result.

[0040] One or more technical solutions provided in the present invention have at least the following technical effects or advantages:

[0041] (1) The present invention adopts a context-based learning-based 3D medical image multi-organ reconstruction method, combining the semantic guidance of the support set and context information, thereby improving the generalization ability and prediction accuracy of the context-based multi-organ segmentation network;

[0042] (2) The present invention utilizes the global context information provided by the image support set to give the model stronger contextual reasoning capabilities, thereby enabling multi-organ reconstruction of downstream three-dimensional medical images without the need for training;

[0043] (3) The present invention utilizes a feature multi-scale cross-fusion module integrated into an encoder-decoder structure with residual connections to effectively integrate information across spatial scales. At the same time, a multi-strategy data enhancement scheme is adopted during the training process. The global context information provided by the support set is used to enable the model to have stronger contextual reasoning capabilities, thereby achieving high-precision multi-organ reconstruction of downstream domain three-dimensional medical images without training. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 A flowchart of the steps of a method for 3D medical image multi-organ reconstruction based on contextual learning provided by an embodiment of the present invention;

[0045] Figure 2 Schematic diagram of a feature multi-scale cross-fusion module provided in an embodiment of the present invention;

[0046] FIG3( a ) is a slice visualization diagram of an abdominal multi-organ CT image provided by an embodiment of the present invention;

[0047] FIG3( b ) is a slice diagram of the segmentation result of the abdominal multi-organ CT image provided by an embodiment of the present invention;

[0048] FIG3( c ) is a final 3D reconstruction of multiple abdominal organs provided by an embodiment of the present invention;

[0049] Figure 4 Schematic diagram of the contextual multi-organ segmentation network provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0050] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts are within the scope of protection of the present invention.

[0051] The present invention provides a method for reconstructing multiple organs from three-dimensional medical images based on contextual learning. Figure 1 As shown, the method includes the following steps S101 to S103.

[0052] S101, obtaining a medical image to be inferred and an image support set, preprocessing the medical image to be inferred to obtain a preprocessed medical image to be inferred, and preprocessing the image support set to obtain a preprocessed support set image; wherein the preprocessed medical image to be inferred includes a plurality of medical image sub-blocks to be inferred, and the preprocessed support set image includes a plurality of image support set sub-blocks;

[0053] Specifically, in step S101, the medical image to be inferred is preprocessed to obtain a preprocessed medical image to be inferred, including: obtaining the medical image to be inferred and an image support set, and preprocessing the medical image to be inferred to obtain a preprocessed medical image to be inferred, including steps S1011 to S1013.

[0054] S1011, converting the data format of the medical image to be inferred and determining whether the medical image to be inferred is complete. If so, resampling the medical image to be inferred to obtain a resampled pre-processed medical image to be inferred; if not, not processing the medical image to be inferred;

[0055] S1012, determining a minimum bounding box of the resampled preprocessed medical image to be inferred, and cropping the resampled preprocessed medical image to be inferred according to the minimum bounding box to obtain a preprocessed sub-block image;

[0056] S1013, normalizing the preprocessed sub-block image to obtain a preprocessed medical image to be inferred.

[0057] For example, the medical images to be inferred include common medical image formats such as DICOM (suffix .dcm), MHD (suffix .mhd and .raw) and NIFTY (suffix .nii or .nii.gz). To facilitate subsequent processing, all raw data are converted into a unified format, including a four-dimensional array and its corresponding compressed file.

[0058] Data integrity checks determine input correctness, including consistency of geometric attributes such as origin, orientation, size, and spacing between the data and labels, as well as any missing values ​​in the label file and whether the label file is continuous and free of outliers. If corruption or outliers are detected, irreparable data is discarded, and partially damaged data is repaired through interpolation or filling. If there are no corrupted data or outliers, subsequent preprocessing operations will proceed.

[0059] S102 inputs the pre-processed medical image to be inferred and the pre-processed support set image into the trained contextual multi-organ segmentation network for zero-sample inference to obtain multiple processed images; Figure 4 As shown, the contextual multi-organ segmentation network consists of an encoder and a decoder connected sequentially.

[0060] The encoder includes: multiple sequentially connected feature multi-scale cross-fusion modules, such as Figure 2 The figure shows a feature multi-scale cross fusion module, and two identical downsampling modules are connected between the two feature multi-scale cross fusion modules.

[0061] The decoder includes: a plurality of sequentially connected feature multi-scale cross-fusion modules, and two identical upsampling modules are connected between two feature multi-scale cross-fusion modules.

[0062] The encoder uses a downsampling module and a multi-scale cross-fusion module to interactively process the preprocessed medical image to be inferred and the preprocessed support set image respectively, and extracts features of the preprocessed medical image to be inferred based on the interactive features to obtain the first extracted features.

[0063] The decoder uses an upsampling module and a feature multi-scale cross-fusion module to restore the feature dimension of the first extracted feature to obtain multiple processed images.

[0064] Specifically, in step S102, the training process of the contextual multi-organ segmentation network includes:

[0065] (1) Preprocessing the original medical image with labels to obtain the preprocessed image;

[0066] Here, the original medical image with labels is preprocessed to obtain the preprocessed image, including:

[0067] (1.1) Obtain the original medical images with labels, and perform data format conversion and integrity check on the original medical images;

[0068] (1.2) cropping the original medical image, retaining the non-zero area of ​​the cropped image, and obtaining a cropped image;

[0069] (1.3) Perform standard normalization on the cropped image to obtain the preprocessed image.

[0070] (2) Divide the preprocessed images into a training set and a test set;

[0071] (3) The contextual multi-organ segmentation network is trained based on the training set and the test set using the back propagation training method to obtain a trained contextual multi-organ segmentation network.

[0072] Here, the contextual multi-organ segmentation network is trained based on the training set and the test set using the back propagation training method to obtain the trained contextual multi-organ segmentation network, including:

[0073] (3.1) Set the training batch, learning rate, and number of iterations;

[0074] (3.2) Calculate the prediction results using the loss function of the contextual multi-organ segmentation network; calculate the loss between the prediction results and the labels;

[0075] (3.3) The contextual multi-organ segmentation network is optimized according to the loss to obtain the trained contextual multi-organ segmentation network.

[0076] For example, step S1: Preprocess the original medical images and labels, including CT, MRI, ultrasound, X-ray, etc. First, convert the dataset format and check the data integrity; then resample to convert the images and labels to a uniform spatial resolution; then crop the input images and labels to retain the non-zero areas; and finally, normalize the images. This preprocessing of the original images and labels improves the final multi-organ reconstruction accuracy and model generalization ability, as specifically implemented as follows:

[0077] Step S11: converting the data set format and checking the data integrity;

[0078] Common medical image formats include DICOM (with the suffix .dcm), MHD (with the suffixes .mhd and .raw), and NIFTY (with the suffix .nii or .nii.gz). To facilitate subsequent processing, all raw data are converted into a unified format, including a four-dimensional array and its corresponding compressed file.

[0079] Data integrity checks determine input correctness, including consistency of geometric attributes such as origin, orientation, size, and spacing between the data and labels, as well as any missing values ​​in the label file and whether the label file is continuous and free of outliers. If corruption or outliers are detected, irreparable data is discarded, and partially damaged data is repaired through interpolation or filling. If there are no corrupted data or outliers, subsequent preprocessing operations will proceed.

[0080] Step S12: resampling the data and labels;

[0081] First, the target resolution is determined by averaging the voxel spacing across all images in the dataset. Then, the images are resampled using cubic spline interpolation to preserve the accuracy of details and edge information. Because labels are discrete category masks, resampling is performed using nearest neighbor interpolation to prevent ambiguity in category labels. This resampling ensures spatial alignment and consistency between images and labels, while simplifying the complexity of processing data from different sources and resolutions.

[0082] Step S13: The input image and label are then cropped to retain the non-zero area;

[0083] The smallest bounding box is found within the 3D medical image and its corresponding label. The extent of this smallest bounding box is determined by the non-zero voxels in both the image and the label, which represent the region of interest or target organ. The invalid regions outside the bounding box, which are all zeros, are then cropped. This aligns the image and label to a unified initial sub-block. While the cropped image and label have no effect on the final segmentation result, reducing their size can significantly improve computational efficiency for some datasets.

[0084] Step S14: Standardize the image.

[0085] When processing three-dimensional voxel data, convolutional neural networks may ignore the size differences of voxels in actual physical space. By resizing image data and labels from different sources or different resolutions, we can ensure that the actual physical space represented by each voxel is of the same size, avoiding the inability of the model to accurately learn spatial features due to resolution differences. Subsequently, the image grayscale values ​​are normalized to make the grayscale value distribution of each input image consistent, while the labels maintain their discrete values ​​unchanged, because the pixel values ​​of the labels represent different category information and cannot be normalized. Z-score-based normalization includes the following steps:

[0086] Step S141: Mark the area to be segmented as the foreground category, collect the intensity information of all foreground voxels, and calculate the mean value of these voxels and standard deviation :

[0087] ;

[0088] ;

[0089] in, The foreground area Voxel value, Indicates the total number of foreground pixels;

[0090] Step S142: Normalize the resampled image data according to the result of step S141:

[0091] ;

[0092] in, Represents the normalized pixel values, represents the mean value of foreground voxels; represents the standard deviation of foreground voxels; that is, voxel intensity normalization of all image data is done by subtracting the foreground voxel mean and dividing by the standard deviation.

[0093] Step S2: performing training division on the pre-processed medical imaging data;

[0094] Step S21: The entire data is divided into a training set and a test set in a ratio of 4:1. The training batch is set to 4, and the first quarter of the data in each training batch is used as the training image, and the remaining three quarters of the image-label pair data is used as the support set. In the preprocessing stage, the images and labels are adjusted to a uniform sub-block, and all training images are randomly cropped or padded based on the central area to generate a resolution of To ensure that each training sub-block contains foreground content, thereby improving the fitting efficiency of network training; the image-label pair sub-block as the support set is scaled to , to ensure that the support set can preserve global information.

[0095] Step S22: Each image in the test set is randomly cropped to obtain a test sub-block. The resolution of the resulting test sub-block is the same as the training sub-block, and the test sub-block does not need to contain labels. This randomness ensures the versatility of the test data. The support set in the test set is scaled to the same size as the test sub-block to preserve global features.

[0096] Specifically, in step S102, the encoder uses a downsampling module and a multi-scale cross-fusion module to interactively process the pre-processed medical image to be inferred and the pre-processed support set image respectively, and extracts features of the pre-processed medical image to be inferred based on the interactive features to obtain a first extracted feature; the decoder uses an upsampling module and a feature multi-scale cross-fusion module to restore the feature dimension of the first extracted feature to obtain multiple processed images; image post-processing is performed based on the multiple processed images to obtain a three-dimensional reconstruction result of multiple organs.

[0097] Specifically, in step S102, the feature multi-scale cross fusion module is expressed as:

[0098] ;

[0099] in, represents the training sub-block; Represents the image support set sub-block; represents the first learnable parameter; represents the second learnable parameter; represents the training sub-block after interaction; Represents the support set block after interaction.

[0100] For example, the training sub-block u and the support set sub-block As input, splicing is performed in the channel dimension so that subsequent convolution can consider the features of both at the same time, and then passing through the cross convolution block to generate interactive feature representation:

[0101] ;

[0102] Among them, A(x) represents the nonlinear activation function, Represents a model with learnable parameters The cross convolutional layers are used to capture the feature interactions between the input image and the support set.

[0103] The mean of the interaction features is calculated for all generated interaction features. Take the mean to generate the updated input image features :

[0104] ;

[0105] This step enables the features of the training sub-block to interact and balance with the features of the support set sub-block, while eliminating the excessive influence of a specific support set on the features of the training sub-block, thereby improving the adaptability and robustness of the model to diverse support sets.

[0106] In the image support set, the support set features after each interaction Extract new support set features through convolutional layers , this feature can capture more contextual information:

[0107] ;

[0108] in, With learnable parameters The convolution operation is used to extract the feature representation after interaction.

[0109] The feature multi-scale cross fusion module enables each support set and input image to interact with other support sets and input images through their average values, extracting information from features at different scales and fusing them:

[0110] ;

[0111] Specifically, in step S102, two identical downsampling modules downsample the medical image to be inferred and the image support set respectively through a multi-layer perceptron; two identical upsampling modules upsample the medical image to be inferred and the image support set respectively through transposed convolution.

[0112] Exemplarily, as shown in Figure 3, the encoder network has six layers, and each layer uses a multi-layer perceptron structure based on three-dimensional convolution to extract features. The input of the network includes training sub-blocks and support set sub-blocks, which are spliced ​​in pairs in the channel dimension and then input into the encoder for feature extraction. The first layer of the encoder consists only of a feature multi-scale cross-fusion module, and the remaining layers consist of a feature multi-scale cross-fusion module and spatial downsampling of the training sub-blocks and support set sub-block representations, thereby achieving efficient feature extraction and representation. Spatial downsampling reduces the spatial resolution of the feature map through 3D convolution operations, while enhancing the depth of the features by increasing the number of convolution layers. In this way, the encoder can gradually extract higher-level features while retaining lower-level detail information, thereby achieving efficient feature extraction and representation.

[0113] The decoder network has five layers in total, and the CNN decoder in the 3D U-Net network is used as the basis of the decoder network. Each layer of the decoder first spatially upsamples the training sub-blocks and the support set sub-blocks through transposed convolution to gradually restore the spatial structure and details of the image; then, the feature map of the decoder and the feature map of the same size in the encoder path are spliced ​​in the channel dimension through jump connections. This jump connection can directly pass the high-level features extracted from the encoder to the decoder, ensuring that the spatial details are preserved and enhanced at high resolution; the spliced ​​feature map will then pass through the feature multi-scale cross-fusion module to effectively fuse information of different scales, so that each feature can fully express the global and local feature details. At the end of the decoder, through a single Convolution maps the fused feature map to the required number of output categories, thus forming the final segmentation prediction map. The entire process ensures the effective fusion of high-resolution features and multi-scale information through gradual upsampling, skip connections, and feature multi-scale cross-fusion modules, thereby improving the accuracy and reliability of the segmentation results.

[0114] The encoder and decoder are cascaded in sequence, and the output of the encoder is fused with the input of the decoder through jump connections at each layer, and high-resolution features are retained through residual connections. By introducing deep supervision, feature interactions between the encoder and decoder are established, thus forming a medical image multi-organ reconstruction network with contextual understanding capabilities. Deep supervision refers to applying loss functions to multiple intermediate layers of the network, which not only supervises the final output but also directly optimizes the feature representation of the intermediate layers, thereby improving gradient flow, improving the network's convergence speed and segmentation accuracy. The loss function of the contextual multi-organ segmentation network is expressed as:

[0115] ;

[0116] in, Indicates the The target loss of the layer; Indicates the total number of decoder layers; represents the medical image to be inferred; represents the image support set; Indicates a label.

[0117] The output of layer 0 has the highest resolution, The layer output has the lowest resolution.

[0118] S103: performing image post-processing based on the multiple processed images to obtain a multi-organ three-dimensional reconstruction result.

[0119] Specifically, in step S103, image post-processing is performed based on the plurality of processed images to obtain a multi-organ three-dimensional reconstruction result, including:

[0120] (1) assigning corresponding weights to the multiple processed images respectively, normalizing the weights of adjacent processed images, and performing weighted summation on all the multiple processed images to obtain the overall prediction result; wherein the weights follow a Gaussian distribution;

[0121] (2) The overall prediction results are processed with Gaussian blur and refined using the connected domain optimization algorithm to obtain the three-dimensional reconstruction results of multiple organs.

[0122] For example, this method implements prediction for 3D medical image multi-organ reconstruction in downstream tasks. The image and support set to be inferred are fed into a trained context-based learning-based 3D medical image multi-organ reconstruction network, which leverages the global context information captured by the network for zero-shot inference. This process enables accurate prediction of data from different domains without requiring training, resulting in the generation of high-quality image sub-blocks.

[0123] The implementation of zero-shot reasoning relies on the extensive contextual understanding acquired by the network during training. This is particularly true for multi-organ reconstruction tasks. By combining multi-level contextual information with support set hints, the network can directly reason without updating weights during the inference process. Because the network has accumulated rich contextual information and generalization capabilities during training, it can directly apply this experience to reasoning tasks in different domains, ensuring cross-domain universality. Because the network can leverage the extensive semantic and structural information acquired during previous training, it can efficiently reason about new image domains and generate accurate segmentation results, significantly improving the efficiency and adaptability of the inference process.

[0124] Assign a Gaussian-distributed weight to each predicted 3D image sub-block, normalize the weights of adjacent predicted sub-blocks, and perform weighted summation on all predicted sub-blocks to obtain the overall prediction result.

[0125] The predicted segmentation results are Gaussian blurred to smooth the edges, and the connected domain optimization algorithm is applied to further refine the results, ultimately generating a complete and accurate three-dimensional multi-organ reconstructed image.

[0126] Specifically, back propagation training is performed on the contextual multi-organ segmentation network that has completed initial training to obtain a trained contextual multi-organ segmentation network.

[0127] Here, the contextual multi-organ segmentation network that has completed the initial training is subjected to back-propagation training to obtain a trained context-based learning three-dimensional medical image multi-organ segmentation network.

[0128] Step 1: First, set the training batch size to 4, use IN as the normalization operator in forward propagation, use the AdamW optimizer, and set the initial learning rate to 0.001. The training process is set to 500 iterations, and the activation function is Leaky ReLU. During training, the support set size for each example is fixed at 4; different numbers of candidates are considered during inference.

[0129] Step 2: After the network weights are forward propagated, the loss between the prediction result and the label is calculated through the loss function;

[0130] Step 3: Repeat step 2 until the loss reaches convergence, then stop training and select the node weight with the highest verification score during training as the final network weight.

[0131] The effects of the present invention can be further illustrated by the following simulation results:

[0132] 1. Simulation conditions

[0133] The multi-organ reconstruction network designed during the simulation process is based on Python 3.10 and Pytorch 2.0.0 framework;

[0134] The machine used to train the reconstructed network during the simulation is equipped with 2 GeForce RTX V100 GPUs;

[0135] The dataset used in the simulation is an abdominal multi-organ CT image and label dataset obtained from the public BTCV dataset.

[0136] 2. Simulation Content

[0137] Under the above simulation conditions, the method of the present invention is used to perform three-dimensional intelligent reconstruction on the obtained abdominal multi-organ CT images and label data sets, wherein:

[0138] Figure 3(a) is a slice visualization of a multi-organ CT image of the abdomen;

[0139] FIG3( b ) is a slice diagram of the segmentation result of the abdominal multi-organ CT image according to the present invention;

[0140] Figure 3(c) shows the final 3D reconstruction of the abdominal multi-organs, which includes 13 regions: spleen, right kidney, left kidney, gallbladder, esophagus, liver, stomach, aorta, inferior vena cava, portal vein and splenic vein, pancreas, right adrenal gland, and left adrenal gland.

[0141] As can be observed in Figure 3(a), the resolution between the organs in the abdominal region is low and the boundaries are blurred. Figure 3(b) demonstrates the segmentation effect of the present invention, which can automatically and accurately segment multiple organs in the abdomen, effectively avoiding the interference of a large amount of blurred background, and the edges between the organs are also clearer. Figure 3(c) further demonstrates the final reconstructed three-dimensional structure of multiple organs in the abdomen, clearly showing the relative position and morphology of each organ.

[0142] The various embodiments in this specification are described in a progressive manner. References to the same or similar parts between the various embodiments are sufficient. Each embodiment focuses on the differences from other embodiments. All or part of the present invention can be used in a variety of general or specialized computer system environments or configurations. For example, personal computers, server computers, handheld or portable devices, tablet devices, mobile communication terminals, multiprocessor systems, microprocessor-based systems, programmable electronic devices, network PCs, minicomputers, mainframe computers, and distributed computing environments that include any of the above systems or devices.

[0143] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, it should be understood by those skilled in the art that the technical solutions described in the aforementioned embodiments may still be modified, or some or all of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the present invention.

Claims

1. A method for 3D medical image multi-organ reconstruction based on contextual learning, characterized in that: include: Acquiring a medical image to be inferred and an image support set, preprocessing the medical image to be inferred to obtain a preprocessed medical image to be inferred, and preprocessing the image support set to obtain a preprocessed support set image; wherein the preprocessed medical image to be inferred includes a plurality of medical image sub-blocks to be inferred, and the preprocessed support set image includes a plurality of image support set sub-blocks; The preprocessed medical image to be inferred and the preprocessed support set image are input into the trained contextual multi-organ segmentation network for zero-sample inference to obtain a plurality of processed images; wherein the contextual multi-organ segmentation network includes an encoder and a decoder connected in sequence; the encoder includes: a plurality of feature multi-scale cross-fusion modules connected in sequence, and two identical downsampling modules are connected between two feature multi-scale cross-fusion modules; the decoder includes: a plurality of feature multi-scale cross-fusion modules connected in sequence, and two identical upsampling modules are connected between two feature multi-scale cross-fusion modules; the encoder uses the downsampling module and the multi-scale cross-fusion module to interactively process the preprocessed medical image to be inferred and the preprocessed support set image respectively, and extracts features from the preprocessed medical image to be inferred based on the interactive features to obtain a first extracted feature; the decoder uses the upsampling module and the feature multi-scale cross-fusion module to restore the feature dimension of the first extracted feature to obtain a plurality of processed images; The feature multi-scale cross fusion module is expressed as: ； in, represents the training sub-block; Represents the image support set sub-block; represents the first learnable parameter; represents the second learnable parameter; represents the training sub-block after interaction; represents the support set block after interaction; The loss function of the contextual multi-organ segmentation network is expressed as: ； in, Indicates the The target loss of the layer; Indicates the total number of decoder layers; represents the medical image to be inferred; represents the image support set; Indicates a label; The encoder uses the downsampling module and the multi-scale cross fusion module to interactively process the preprocessed medical image to be inferred and the preprocessed support set image respectively, and extracts features from the preprocessed medical image to be inferred based on the interactive features to obtain first extracted features, including: Splicing the preprocessed medical image to be inferred and the preprocessed support set image in the channel dimension to obtain a plurality of spliced ​​images; Performing cross convolution processing on the multiple stitched images respectively to obtain initial interaction features corresponding to each stitched image; Performing mean calculation on the initial interaction features corresponding to each spliced ​​image to obtain the interaction features corresponding to the preprocessed medical image to be inferred; Resampling the pre-processed medical image to be inferred using the interactive feature and the downsampling module to obtain a first extracted feature; Image post-processing is performed based on the multiple processed images to obtain a three-dimensional reconstruction result of multiple organs.

2. The context-based learning-based 3D medical image multi-organ reconstruction method according to claim 1, characterized in that: The obtaining of the medical image to be inferred and the image support set, and preprocessing the medical image to be inferred to obtain the preprocessed medical image to be inferred, includes: Converting the data format of the medical image to be inferred and determining whether the medical image to be inferred is complete; if so, resampling the medical image to be inferred to obtain a resampled preprocessed medical image to be inferred; if not, processing the medical image to be inferred; Determining a minimum bounding box of the resampled preprocessed medical image to be inferred, and cropping the resampled preprocessed medical image to be inferred according to the minimum bounding box to obtain a preprocessed sub-block image; The preprocessed sub-block image is standardized to obtain a preprocessed medical image to be inferred.

3. The context-based learning-based 3D medical image multi-organ reconstruction method according to claim 1, characterized in that: The training process of the contextual multi-organ segmentation network includes: Preprocessing the original medical image with labels to obtain a preprocessed image; Dividing the preprocessed images into a training set and a test set; The contextual multi-organ segmentation network is trained according to the training set and the test set using a back-propagation training method to obtain a trained contextual multi-organ segmentation network.

4. The context-based learning-based 3D medical image multi-organ reconstruction method according to claim 3, characterized in that: The preprocessing of the original medical image with the label to obtain the preprocessed image includes: Obtaining an original medical image with a label, performing data format conversion and integrity check on the original medical image, and resampling the original medical image with a label to obtain resampled data; Clipping the resampled data, retaining a non-zero region of the clipped image, and obtaining a clipped image; Performing standard normalization on the cropped image to obtain a preprocessed image.

5. The context-based learning-based 3D medical image multi-organ reconstruction method according to claim 3, characterized in that: The training method using back propagation is used to train the contextual multi-organ segmentation network according to the training set and the test set to obtain a trained contextual multi-organ segmentation network, including: Set the training batch, learning rate, and number of iterations; Calculating a prediction result using the loss function of the contextual multi-organ segmentation network; calculating the loss between the prediction result and the label; The contextual multi-organ segmentation network is optimized according to the loss to obtain a trained contextual multi-organ segmentation network.

6. The context-based learning-based 3D medical image multi-organ reconstruction method according to claim 1, characterized in that: The two identical downsampling modules downsample the medical image to be inferred and the image support set respectively through a multi-layer perceptron; the two identical upsampling modules upsample the medical image to be inferred and the image support set respectively through transposed convolution.

7. The method for 3D medical image multi-organ reconstruction based on contextual learning according to claim 1, characterized in that: The performing image post-processing according to the plurality of processed images to obtain a multi-organ three-dimensional reconstruction result includes: Assigning corresponding weights to the multiple processed images respectively, normalizing the weights of adjacent processed images, and performing weighted summation on all the multiple processed images to obtain an overall prediction result; wherein the weights obey a Gaussian distribution; The overall prediction result is subjected to Gaussian blur processing and refined using a connected domain optimization algorithm to obtain a multi-organ three-dimensional reconstruction result.

Images