Multimodal medical image rectal tumor staging automatic 3d fine annotation system

Abstract

The application discloses a kind of multi-modal medical image rectal tumor staging automatic 3D fine annotation system, including image pre-processing module, image 3D feature extraction module, fine 3D annotation training dataset, annotation model and model self-modification module, by constructing and training based on unlabelled multi-modal medical image data 3D feature extraction big model, mining and extracting the 3D feature of different modal medical image medium, design knowledge-guided rectal tumor staging 3D fine annotation model is segmented and detected to rectum and lymph node lesion, preliminary sketch out the 3D position structure of rectal tumor and pathological lymph, give T staging and N staging diagnostic conclusion, finally, diffusion generation technology is used to self-modification to 3D annotation model, accurately sketch out the 3D position structure of rectal tumor and pathological lymph.

Description

Technical Field

[0001] This invention relates to the field of medical image processing technology, and in particular to an automatic 3D fine annotation system for staging rectal tumors in multimodal medical images. Background Technology

[0002] Rectal tumors are among the most common malignant tumors of the digestive system. In recent years, with the improvement of people's living standards and changes in dietary structure, the incidence of rectal tumors has been increasing year by year. Rectal tumor examination mainly uses technologies such as computed tomography (CT) and magnetic resonance imaging (MRI). However, high-resolution MRI is expensive, has a long scanning time, and is highly sensitive to intestinal peristalsis and respiration. Furthermore, MRI scans are performed in a completely closed environment, which can lead to poor imaging results in some patients with poor tolerance. In addition, MRI and CT generate video data, resulting in large image data volumes. Misdiagnosis and underdiagnosis of T and N staging due to diagnostic fatigue and lack of experience among physicians are common, and work efficiency is low. Therefore, researching and implementing an automatic 3D fine-annotation system for rectal tumor staging based on multimodal medical imaging such as MRI and CT has significant research significance and practical clinical application value.

[0003] Patent application number CN202310618617.9 discloses a method for identifying lung tumor regions based on imaging features. The method determines the lung tumor region by the difference between the first feature of the first image and the second feature of the second image, which improves the efficiency and accuracy of lung tumor region analysis. However, it only delineates the location of the lung tumor on two-dimensional (2D) CT medical images. Furthermore, due to the unique nature of medical images, annotation tasks require specialized medical knowledge and experience, making the annotation process more cumbersome and time-consuming. Existing databases typically struggle to provide sufficiently large and high-quality 3D annotation datasets for rectal tumor staging. Simultaneously, for multimodal medical images such as CT and MRI, physicians exhibit subjective judgments and individual differences in tumor boundary delineation, leading to potentially inconsistent tumor lesion annotation results. Therefore, the heterogeneity of multimodal medical image data and the inconsistency of manual annotation further complicate the automatic 3D annotation of rectal tumor staging. Additionally, the varying data volume for different T and N stages in the annotation dataset results in an imbalanced sample distribution for rectal tumor T and N stages. This imbalance hinders the training and validation of deep learning algorithms, making it difficult to achieve satisfactory automatic 3D annotation results for rectal tumor staging.

[0004] This invention performs 3D fine-grained annotation of rectal tumor staging on CT and MRI multimodal medical imaging data, automatically delineates the 3D location structure of rectal tumors and diseased lymph nodes, and provides T-staging and N-staging diagnostic conclusions. It can be used for automated assisted diagnosis, as well as for precise individualized surgery, treatment plan decision-making, and patient treatment effect evaluation, and has a wider range of practical medical clinical application value. Summary of the Invention

[0005] The purpose of this invention is to address the current difficulties in obtaining sufficiently large and high-quality 3D annotated datasets for rectal tumor staging, the heterogeneity of multimodal medical image data, the inconsistency of manual annotation, and the imbalance of different T-stage and N-stage data. Therefore, this invention proposes an automatic 3D fine annotation system for rectal tumor staging in multimodal medical images.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] An automatic 3D fine annotation system for staging rectal tumors in multimodal medical images includes an image preprocessing module, an image 3D feature extraction module, a fine 3D annotation training dataset, an annotation model, and a model self-correction module;

[0008] The image preprocessing module includes a data format conversion module, a data reading module, and an image enhancement module. Through data format conversion, data reading, and image enhancement operations, it transforms multimodal raw medical image data into a unified format applicable to all functional modules of the system.

[0009] The image 3D feature extraction module includes two parts: an image encoder and an image decoder. It adopts self-encoding technology to randomly cover the image and uses the completion of the covered image as a learning target to extract 3D medium features from the input multimodal unlabeled medical video.

[0010] The finely annotated 3D training dataset contains a multimodal medical image set with finely annotated 3D images of rectal tumors and lesion lymph nodes at different T and N stages;

[0011] The annotation model includes a T-staging task-guided 3D annotation module for rectal tumors and an N-staging task-guided 3D annotation module for lesion lymph nodes. Based on the 3D media features extracted by the image 3D feature extraction module, the preliminary results of T and N-staging 3D annotation for rectal tumors are obtained by training the rectal tumor 3D annotation model and the lesion lymph node 3D annotation model.

[0012] The model self-correction module includes a diffusion generation model, an annotation information processing module, and a supervised learning module, which completes the parameter correction of the annotation model and obtains the 3D fine annotation results of T and N staging of rectal tumors.

[0013] Specifically, the data format conversion module combines the pixel values ​​of each frame of the DICOM format medical image video into 3D volume data according to their spatial positions to obtain voxel size and spacing information, construct the header information of the NiFTI file and save it.

[0014] The data reading module uses a medical image processing library based on a deep learning framework to read the converted NiFTI data and establish a mapping relationship between the original image data and the label data.

[0015] The image enhancement module performs operations such as image resampling, noise reduction, random affine transformation, and adding channels to the image size to enhance the established mapping relationship between image data and label data.

[0016] The image 3D feature extraction module includes an image encoder and an image decoder, both of which contain multi-layer networks;

[0017] The image encoder divides each frame of the input raw unlabeled multimodal medical video into regular and non-overlapping small image blocks. It randomly samples one image block from all the image blocks and marks it as "unmasked". It marks the remaining unsampled image blocks as "masked". The random sampling is repeated several times. Then, it encodes the coordinate position information of all the randomly masked image blocks in the image. All image blocks in each image data are linearly arranged to obtain the image block vector code.

[0018] The image decoder completes the image information for the image blocks marked as "masked" by using the input image block vector and the decoder network weights;

[0019] The image 3D feature extraction module uses the same training strategy for both rectal tumors and lesions in lymph nodes to extract 3D features from different modalities of image media.

[0020] Specifically, the T-staging task-guided 3D annotation module for rectal tumors includes a rectal tumor segmentation module and a rectal tumor T-staging diagnosis module;

[0021] The rectal tumor segmentation module mainly includes a multi-layer segmentation structure and a skip connection mechanism composed of image convolution, downsampling and upsampling operations.

[0022] The multi-layer segmentation structure first undergoes image convolution and downsampling operations, using the image medium information output by the image 3D feature extraction module to set pre-trained weights such as parameter initialization, and then undergoes image convolution and upsampling operations to restore the resolution of the feature map to the original image size and generate 3D segmentation prediction results.

[0023] The skip connection mechanism connects the feature map of the last downsampling process with the feature map of the first upsampling process, the feature map of the penultimate downsampling process with the feature map of the second upsampling process, and so on, in order to prevent potential image feature loss due to downsampling.

[0024] The rectal tumor segmentation module adjusts the parameters of the image convolution and downsampling network and the image convolution and upsampling network through the combined action of a multi-layer segmentation structure and a skip connection mechanism until the segmentation accuracy reaches the preset requirements, and then adds the rectal tumor location information.

[0025] Specifically, the rectal tumor T-staging diagnostic module mainly comprises three weight matrices W Q W K W V The self-attention network and the multi-layer neural network for the T-stage task are composed of these components.

[0026] The self-attention network utilizes the image medium information output by the image 3D feature extraction module to initialize the weight matrix and set parameters. The self-attention network primarily uses the following attention formula for pre-training with a large number of 3D images to capture the image medium information. After pre-training, the matrix parameters are initialized and set:

[0027]

[0028] Where X is an image with location markers;

[0029] Q = X * W Q K T For X*W K The transpose of the matrix;

[0030] V = X * W V d K Let K be the dimension of each row vector;

[0031] softmax is the activation function;

[0032] The T-stage task multilayer neural network consists of multiple layers of weights. The input data is multiplied by the weights of each layer to obtain the preliminary T-stage result. When training the module, the process is continuously iterated based on the difference between the generated T-stage information and the T-stage information annotated in the fine 3D labeled training dataset.

[0033] The rectal tumor T-staging diagnostic module adjusts the parameters of the self-attention network and the multi-layer neural network for the T-staging task until the accuracy of the T-staging reaches the preset requirements. Then, it adds the annotation of rectal tumor T-staging diagnostic information and finally completes the 3D fine annotation of rectal tumor T-staging in medical images.

[0034] Specifically, the N-staging task-guided lesion lymph node 3D annotation module includes a lesion lymph node detection module and a lesion lymph node N-staging diagnosis module;

[0035] The lesion lymph node detection module includes a candidate region generation network and a classification and screening mechanism;

[0036] The candidate region generation network is mainly composed of image convolution and downsampling operations. First, after image convolution, the image medium information output by the image 3D feature extraction module is used to set the pre-trained weights such as parameter initialization. Then, after downsampling, a low-resolution image is obtained and image features are extracted. Based on the image features, several 3D candidate regions suspected to be lymph nodes are obtained.

[0037] The classification and screening mechanism classifies each candidate region through a classifier and introduces a non-maximum suppression mechanism to calculate the intersection-union ratio (IUR) of the candidate region with the highest classification score and other candidate regions, and eliminates regions with excessively high IUR to reduce overlapping candidate regions.

[0038] The lesion lymph node detection module, through the combined action of the candidate region generation network and the classification screening mechanism, compares the 3D location information of the candidate region with the finely labeled 3D lesion lymph node locations in the training set, and continuously adjusts the classification network and hyperparameters. When the similarity between the output lesion lymph node 3D location information and the finely labeled lesion lymph node 3D location reaches a preset standard, it determines whether it is a lesion lymph node region.

[0039] Specifically, the N-stage diagnostic module for lesion lymph nodes has the same architecture as the T-stage diagnostic module for rectal tumors;

[0040] The N-staging diagnostic module for lesions in lymph nodes includes a self-attention network and a multi-layer neural network for the N-staging task. It completes the N-staging task by using the location information of the lesions in lymph nodes, the training set, and the features of the image medium. Finally, it integrates the N-staging information and the 3D location information of the lesions in lymph nodes output by the lesion lymph node detection module to complete the 3D fine annotation of the N-staging of lesions in lymph nodes in medical images.

[0041] Specifically, the annotation information processing module uses the 3D location information of rectal tumors and lesion lymph nodes to set the pixel value of the rectal tumor and lesion lymph node area to 255, and the pixel value of other areas to 0 and saves it to obtain a location annotation image;

[0042] The diffusion generation model includes a neural network module for optimization iteration. During the generation process, starting from a known initial state, it introduces some noise or randomness through multiple iterations. The introduction of noise causes the labeled image to have small changes at each step, which gradually generates more complex and realistic samples.

[0043] The forward process mainly defines the relationship between the next time step and the previous time step, obtaining the image relationship from the noise-added image to any time step t:

[0044]

[0045] Where, x t Let x represent the image at time t, x0 represent the image at time t=0, ε represent noise, and α represent... t =1-β t , α t α t-1 α t-2 α t-3 …α2α1;

[0046] β is a number between [0,1] used to generate the coefficients of ε and x;

[0047] The reverse process refers to the goal starting from the last x. t From the noisy image at time x0, the original image at time x0 is recovered, as expressed by the following formula for any time x. t All of these can be considered as being obtained by directly adding noise to a certain x0 original image:

[0048]

[0049] Where P represents a probability distribution, and the distribution from x0 to x is known. t By adding noise ε, we can obtain its previous time step x. t-1 The probability distribution;

[0050] The diffusion generation model ultimately guides the generation of medical images for T and N stages with a smaller number of specified data samples through a neural network module, thereby expanding the sample size.

[0051] Specifically, the supervised learning module incorporates the data generated by the diffusion generation model into the fine-grained 3D annotation training dataset to expand the existing fine-grained annotation dataset. The expanded dataset is then input into the annotation model to train the network again, thereby improving the accuracy of the network's 3D annotation of T and N stages in medical images.

[0052] Compared with the prior art, the beneficial effects of the present invention are:

[0053] 1. This invention proposes a knowledge-guided 3D fine-annotation model for rectal tumor staging. This model utilizes a small dataset of finely annotated 3D rectal tumor staging data and uses T-staging and N-staging task knowledge to guide the fine-tuning of the 3D rectal tumor staging annotation model. It fully leverages multimodal medical imaging media information to effectively capture spatial information, thereby better understanding the morphology and distribution of rectal tumors and diseased lymph nodes. It preliminarily outlines the 3D location structure of rectal tumors and diseased lymph nodes, and provides T-staging and N-staging diagnostic conclusions. This effectively reduces the differences caused by the subjective bias of doctors during the annotation process, making the annotation results more accurate and consistent.

[0054] 2. This invention proposes a self-correction method for a 3D fine-annotation model of rectal tumor staging based on diffusion generation technology. This method accurately delineates the 3D location structure of rectal tumors and diseased lymph nodes. By supplementing samples of different T and N stages through generation technology, the data proportions of each T and N stage are made similar, thereby enhancing the robustness of the model in handling imbalanced data, avoiding the influence of imbalanced training data, helping to ensure that the model better learns and understands the characteristics of each stage, avoiding over-reliance on a small number of tumor stage data, and improving the accuracy of 3D annotation of rectal tumor staging.

[0055] 3. This invention utilizes a supervised learning module within a self-correction module to incorporate massive amounts of unlabeled multimodal medical image data into a fine 3D labeled training dataset after system training. This expands the existing fine labeled dataset and solves the problem of current deep learning models for medical images lacking large-scale labeled data for effective supervised learning. The expanded dataset is then input into the labeling model to retrain the network, thereby improving the accuracy of the network's 3D labeling of T and N stages in medical images. Attached Figure Description

[0056] To illustrate the technical solutions in the embodiments of the present invention or the prior art more specifically and intuitively, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below.

[0057] Figure 1 This is a system flowchart illustrating the overall operation of the present invention;

[0058] Figure 2 This is a schematic diagram of the system functional structure modules of the present invention;

[0059] Figure 3 This is a flowchart of the image 3D feature extraction module of the present invention;

[0060] Figure 4 This is a flowchart of the T-staging task-guided 3D annotation module for rectal tumors according to the present invention;

[0061] Figure 5This is a flowchart of the N-stage task-guided lesion lymph node 3D annotation module of the present invention;

[0062] Figure 6 This is a flowchart of the self-correction module based on the diffusion generation model of the present invention;

[0063] Figure 7 This is a visual diagram illustrating the system functions of the present invention. Detailed Implementation

[0064] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0065] Reference Figure 1 - Figure 7 An automatic 3D fine annotation system for staging rectal tumors in multimodal medical imaging includes an image preprocessing module, an image 3D feature extraction module, a fine 3D annotation training dataset, an annotation model, and a model self-correction module.

[0066] Among them, the fine 3D annotation training dataset contains a multimodal medical image set with fine 3D annotations of rectal tumors and lesion lymph nodes at different T and N stages;

[0067] The image preprocessing module transforms multimodal raw medical image data into a unified format suitable for all functional modules of the system through data format conversion, data reading, and image enhancement operations. The image preprocessing module includes the following steps to ensure data consistency and improve the performance of subsequent deep learning models:

[0068] (1) Data format conversion:

[0069] When processing 3D medical image data, a DICOM reader library or tool is used to extract each frame of the 2D image from the DICOM format medical video. The pixel values ​​of each 2D image are combined into 3D volume data according to their spatial positions. This is done by stacking the pixel values ​​of each frame according to their positions in the DICOM sequence. The voxel size and voxel spacing are extracted from the DICOM sequence. Using the spatial information in the 3D volume data and the obtained voxel size and spacing information, the header information of the NiFTI file is constructed to ensure that the spatial information, including the origin and orientation matrix, is correctly mapped. Finally, the constructed NiFTI header information and the 3D volume data are saved as a NiFTI format file to ensure that the original medical image data format is consistent with the corresponding annotation label data format, providing standard input for subsequent processing.

[0070] (2) Data reading:

[0071] Using a medical image processing library based on a deep learning framework, the converted NiFTI data is read. The medical image data and labeled data are transformed into tensor data structures that can be processed by the deep learning framework, and a mapping relationship between the original image data and the labeled data is established. The data is then integrated into the deep learning model for effective training.

[0072] (3) Image enhancement:

[0073] Image enhancement is achieved by performing operations such as image resampling, noise reduction, random affine transformation, and increasing the number of channels in the image size, based on the established mapping relationship between image data and label data. These operations help improve image quality, increase the robustness of the model to different scenes, and provide more informative image inputs for deep learning network models.

[0074] The above preprocessing steps for 3D medical video can effectively transform the raw medical video data into a format suitable for deep learning models, and improve the quality of the data through image enhancement operations, laying the foundation for subsequent model training.

[0075] The image 3D feature extraction module includes an image encoder and an image decoder. It employs autoencoder technology to randomly occlude the image and uses the occluded image completion as a learning objective to extract 3D media features from the input multimodal unlabeled medical video. The extraction steps of the image 3D feature extraction module are as follows:

[0076] (1) Input the unlabeled multimodal medical video (3D) data output by the image preprocessing module as the training dataset into the image 3D feature extraction module;

[0077] (2) The image decoder consists of a multi-layer network, each layer of which is composed of a one-dimensional weight matrix of different sizes. The total number of layers and the size of the one-dimensional weight matrix of each layer are determined according to the parameters of the training dataset. The input data is encoded by the image encoder, which divides each frame of the training set into regular and non-overlapping small image blocks. One image block is randomly sampled from all the image blocks and marked as "unoccluded", while the remaining unsampled image blocks are marked as "occluded". The random sampling is repeated, and the sampled block is marked as "unoccluded" several times. The number of repetitions of this operation is determined by the image size of the training dataset, maintaining a high "occlusion" ratio to eliminate redundancy in the "unoccluded" data.

[0078] (3) Add the coordinate position information of the image block in the image to all image blocks that have been randomly covered, and then arrange all image blocks in each image data in a linear manner to obtain the image block vector code, including the original image information of each "uncovered" image block, the image block position information, and the "covered" and "uncovered" markers of the image block.

[0079] (4) The image decoder completes the image information of the image blocks marked as “occluded” by input image block vector and decoder network weights, and continuously adjusts the decoder network weights. In the repeated training process, the decoder is guided to complete the image information of all “occluded” image blocks with high fidelity, so as to improve the module’s extraction effect of 3D media features of multimodal medical images.

[0080] (5) The image 3D feature extraction module uses the same training strategy for rectal tumors and lesions of lymph nodes to extract 3D features of different modal image media.

[0081] The image 3D feature extraction module not only completed the "masked" image completion task, but also preserved the image medium information of the medical image in the image decoder network weights of this module. To complete the multimodal medical image staging annotation task, the T-staging task-guided rectal tumor 3D annotation module and the N-staging task-guided lesion lymph node 3D annotation module utilize finely annotated multimodal 3D medical image datasets, combined with the multimodal medical image medium information obtained from the image 3D feature extraction module, to complete the lesion tissue segmentation and detection tasks, as well as the T and N-staging diagnosis tasks of the lesion tissue.

[0082] The annotation model includes a T-staging task-guided 3D annotation module for rectal tumors and an N-staging task-guided 3D annotation module for lesion lymph nodes. The framework of the T-staging task-guided 3D annotation module for rectal tumors is as follows:

[0083] (1) As Figure 4 As shown, the T-staging task-guided 3D annotation module for rectal tumors consists of two parts: a rectal tumor segmentation module and a rectal tumor T-staging diagnosis module. During the construction of this module, a multimodal medical image set with fine 3D annotation of rectal tumor staging will be used as the training dataset.

[0084] (2) Rectal tumor segmentation module: The medical image video data to be labeled is input into the segmentation module after image preprocessing. The segmentation module first outputs a low-resolution image through image convolution and downsampling. This is a multi-layer structure composed of a series of convolution and downsampling (such as max pooling) operations. Each layer consists of multiple convolutions and one max pooling operation. The image medium information output by the image 3D feature extraction module is used to set the pre-trained weights such as parameter initialization. Then, the resolution of the feature map is restored to the original image size through image convolution and upsampling. This is a multi-layer structure composed of a series of convolution and upsampling (such as deconvolution) operations. Each layer consists of multiple convolutions and one deconvolution operation. The module generates 3D segmentation prediction results.

[0085] (3) To prevent potential image feature loss due to downsampling, a skip connection mechanism is introduced: the last downsampling process is connected to the feature map of the first upsampling process, the penultimate downsampling process is connected to the feature map of the second upsampling process, and so on. During training, the differences between the rectal tumor 3D location information generated by the model and the rectal tumor 3D location information finely annotated in the training set are continuously iterated to adjust the parameters of the image convolution and downsampling network and the image convolution and upsampling network until the segmentation accuracy reaches the preset requirements.

[0086] (4) Rectal Tumor T-Stage Diagnosis Module: The medical image video data to be labeled, after image preprocessing, is input together with the 3D location information of the rectal tumor output by the rectal tumor segmentation module into a system consisting of three weight matrices W. Q W K W V The self-attention network, composed of these components, utilizes the image medium information output by the 3D feature extraction module to initialize the weight matrix and set the parameters. The self-attention network primarily employs the following attention formula, pre-training with a large number of 3D images to capture the image medium information. After pre-training, the matrix parameters are initialized:

[0087]

[0088] Where X is an image with location markers;

[0089] Q = X * W Q K T For X*W K The transpose of the matrix;

[0090] V = X * W V d K Let K be the dimension of each row vector;

[0091] softmax is the activation function;

[0092] The output of the self-attention network is summed with the 3D location information of the rectal tumor, and then fed into a multilayer neural network for T-staging, which consists of multiple layers of weights. The input data is multiplied by the weights of each layer to obtain a preliminary T-staging result. During training, the parameters of the self-attention network and the multilayer neural network for T-staging are iteratively adjusted based on the difference between the generated T-staging information and the T-staging information labeled in the training set data, until the T-staging accuracy reaches the preset requirements. Finally, the output integrates the T-staging information from the multilayer neural network for T-staging and the 3D location information of the rectal tumor from the rectal tumor segmentation module, completing the fine 3D annotation of rectal tumor T-staging in medical images.

[0093] The framework of the N-staging task-guided 3D annotation module for lesion lymph nodes is as follows:

[0094] (1) As Figure 5 As shown, the N-staging task-guided lesion lymph node 3D annotation module consists of a lesion lymph node detection module and a lesion lymph node N-staging diagnosis module; this module will use a multimodal medical image set with fine 3D annotation of rectal tumor staging as the training dataset;

[0095] (2) Lymph node detection module: The medical image video data to be labeled is preprocessed and then input into the lymph node candidate region generation network. This network is a multi-layer structure composed of a series of convolution and downsampling (such as max pooling) operations. Each layer consists of multiple convolution and one max pooling operation. This network uses the image medium information output by the multimodal medical image 3D feature extraction module to set the pre-trained weights such as parameter initialization, and uses downsampling to obtain low-resolution images, extracts image features, and obtains several 3D candidate regions of suspected lymph nodes based on the image features.

[0096] (3) For each lymph node candidate region output by the candidate region generation network, the 3D location information is combined with the original resolution image and the image features within the region are extracted by convolution operation to obtain the feature matrix of each candidate region box. When the classifier performs classification calculation on the candidate boxes, a non-maximum suppression mechanism is introduced. The intersection-union ratio (i.e., the ratio of the intersection area of ​​candidate boxes to the union area) is calculated between the candidate box with the highest classification score (i.e., the target most suspected to be a lymph node region) and other candidate boxes. Regions with excessively high intersection-union ratios are eliminated to reduce overlapping candidate boxes.

[0097] (4) When training the lesion lymph node detection module, the 3D location information of the candidate region is compared with the finely annotated 3D lesion lymph node location in the training set. The classification network and hyperparameters (such as candidate box size and intersection-union threshold) are continuously adjusted until the similarity between the output lesion lymph node location information and the finely annotated lesion lymph node location reaches the preset standard, and it is determined whether it is a lesion lymph node region.

[0098] (5) Lymph node N-staging diagnostic module: Its architecture is consistent with that of the rectal tumor T-staging diagnostic module. It consists of a self-attention network and a multi-layer neural network for the N-staging task. It completes the staging task by using the location information of the lesion lymph nodes, the training set and the image medium features. The model is trained and the parameters are adjusted separately for the T and N-staging tasks to adapt to different task scenarios. Finally, the N-staging information output by the multi-layer neural network of the N-staging task and the 3D location information of the lesion lymph nodes output by the lesion lymph node detection part are output to complete the 3D fine annotation of the N-staging of lesion lymph nodes in medical images.

[0099] The model self-correction module includes a diffusion generation model, an annotation information processing module, and a supervised learning module. It completes the parameter correction of the annotation model and obtains the 3D fine-grained annotation results for T and N staging of rectal tumors. The framework of the model self-correction module is as follows:

[0100] (1) Guided diffusion generation model: Based on the 3D location information and staging annotation information of rectal tumors and lesion lymph nodes, guide the diffusion generation model to generate medical images of T and N stages with a small number of specified data samples, expand the number of samples, and reduce the problem of unbalanced distribution of training data of different stages.

[0101] (2) Labeling information processing module: Using the 3D location information of rectal tumors and lesion lymph nodes, the location labeling image is obtained. The specific operation method is to set the pixel value of the rectal tumor and lesion lymph node area to 255, and the pixel value of other areas to 0, and save it; Using the T stage labeling information, the stage text description is created. For example, if the stage is T3, it is converted into the text description language "This is a T3 stage rectal tumor lesion" and saved; For text input, the language model is first used to convert it into an embedding, and then it is mapped into the network through the attention mechanism, that is, the text-guided diffusion generation model is used for diffusion;

[0102] (3) The diffusion generation model includes a neural network module for optimization iteration. In the generation process, it starts from a known initial state and iterates multiple times. Each iteration introduces some noise or randomness. The introduction of noise causes the labeled image to change slightly at each step, so that it gradually generates more complex and realistic samples.

[0103] The forward process mainly defines the relationship between the next time step and the previous time step, obtaining the image relationship from the noise-added image to any time step t:

[0104]

[0105] Where, x t Let x represent the image at time t, x0 represent the image at time t=0, ε represent noise, and α represent... t =1-β t , α t α t-1 α t-2 α t-3 …α2α1;

[0106] β is a number between [0,1] used to generate the coefficients of ε and x;

[0107] The reverse process refers to the goal starting from the last x. t From the noisy image at time x0, the original image at time x0 is recovered, as expressed by the following formula for any time x. t All of these can be considered as being obtained by directly adding noise to a certain x0 original image:

[0108]

[0109] Where P represents a probability distribution, and the distribution from x0 to x is known. t By adding noise ε, we can obtain its previous time step x. t-1 The probability distribution;

[0110] (4) The diffusion generation model trains a neural network module, inputting x. t The image at time step x0 is used to predict the noise added to this image relative to the original image at a certain x0. In other words, the neural network model outputs the noise ε. Based on this noise, the probability distribution of the previous time step is obtained. This probability distribution is then used for random sampling to obtain an image from the previous time step. This previous image is then fed back into the neural network model to predict its noise ε, and this process is repeated until x0 is obtained; while the initial x... t The image is approximately equal to the noise ε, that is, x t Images at any given time can be sampled arbitrarily using a standard normal distribution; ultimately, this leads to the generation of medical images for T and N stages with a smaller number of specified data samples, thus expanding the sample size.

[0111] (5) Supervised learning module: The data generated by the diffusion generation model is incorporated into the fine 3D annotation training dataset to expand the existing fine annotation dataset. The expanded dataset is then input into the annotation model to train the network again, so as to improve the accuracy of the network in 3D annotation of T and N stages of medical images.

[0112] The present invention provides an automatic 3D fine annotation system for rectal tumor staging in multimodal medical imaging. It is used for multimodal medical image video data such as CT and MRI. By constructing and training a large 3D feature extraction model based on unannotated multimodal medical image data, it mines and extracts 3D features of different modalities of medical image media. It designs a knowledge-guided 3D fine annotation model for rectal tumor staging to segment and detect rectal and lymph node lesions, initially delineates the 3D location structure of rectal tumors and diseased lymph nodes, and gives T-staging and N-staging diagnostic conclusions. Finally, it uses diffusion generation technology to self-correct the 3D annotation model and accurately delineates the 3D location structure of rectal tumors and diseased lymph nodes.

[0113] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A multimodal medical imaging system for automatic 3D fine annotation of rectal tumor staging, characterized in that: It includes an image preprocessing module, an image 3D feature extraction module, a fine 3D labeled training dataset, a labeled model, and a model self-correction module; The image preprocessing module includes a data format conversion module, a data reading module, and an image enhancement module. Through data format conversion, data reading, and image enhancement operations, it transforms multimodal raw medical image data into a unified format applicable to all functional modules of the system. The image 3D feature extraction module includes two parts: an image encoder and an image decoder. It adopts self-encoding technology to randomly cover the image and uses the completion of the covered image as a learning target to extract 3D medium features from the input multimodal unlabeled medical video. The finely annotated 3D training dataset contains a multimodal medical image set with finely annotated 3D images of rectal tumors and lesion lymph nodes at different T and N stages; The annotation model includes a T-staging task-guided 3D annotation module for rectal tumors and an N-staging task-guided 3D annotation module for lesion lymph nodes. Based on the 3D media features extracted by the image 3D feature extraction module, the preliminary results of T and N-staging 3D annotation for rectal tumors are obtained by training the rectal tumor 3D annotation model and the lesion lymph node 3D annotation model. The model self-correction module includes a diffusion generation model, an annotation information processing module, and a supervised learning module, which completes the parameter correction of the annotation model and obtains 3D fine annotation results of T and N staging of rectal tumors. The image 3D feature extraction module includes an image encoder and an image decoder, both of which contain multi-layer networks; The image encoder divides each frame of the input raw unlabeled multimodal medical video into regular and non-overlapping small image blocks. It randomly samples one image block from all the image blocks and marks it as "unmasked". It marks the remaining unsampled image blocks as "masked". The random sampling is repeated several times. Then, it encodes the coordinate position information of all the randomly masked image blocks in the image. All image blocks in each image data are linearly arranged to obtain the image block vector code. The image decoder completes the image information for the image blocks marked as "occluded" by using the input image block vector and the decoder network weights; The image 3D feature extraction module uses the same training strategy for both rectal tumors and lesions in lymph nodes to extract 3D features from different modalities of image media.

2. The automatic 3D fine annotation system for multimodal medical imaging staging of rectal tumors according to claim 1, characterized in that: The data format conversion module combines the pixel values ​​of each frame of the DICOM format medical image video into 3D volume data according to their spatial position to obtain its voxel size and spacing information, construct the header information of the NiFTI file and save it. The data reading module uses a medical image processing library based on a deep learning framework to read the converted NiFTI data and establish a mapping relationship between the original image data and the label data. The image enhancement module performs image resampling, noise reduction, random affine transformation, and adds channels to the image size based on the established mapping relationship between image data and label data, thereby enhancing the image.

3. The automatic 3D fine annotation system for multimodal medical imaging staging of rectal tumors according to claim 1, characterized in that: The T-staging task-guided 3D annotation module for rectal tumors includes a rectal tumor segmentation module and a rectal tumor T-staging diagnosis module. The rectal tumor segmentation module mainly includes a multi-layer segmentation structure and a skip connection mechanism composed of image convolution, downsampling and upsampling operations. The multi-layer segmentation structure first undergoes image convolution and downsampling operations, using the image medium information output by the image 3D feature extraction module to set parameters and initialize pre-training weights. Then, it undergoes image convolution and upsampling operations to restore the resolution of the feature map to the original image size and generate 3D segmentation prediction results. The skip connection mechanism connects the feature map of the last downsampling process with the feature map of the first upsampling process, the feature map of the penultimate downsampling process with the feature map of the second upsampling process, and so on, in order to prevent potential image feature loss due to downsampling. The rectal tumor segmentation module adjusts the parameters of the image convolution and downsampling network and the image convolution and upsampling network through the combined action of a multi-layer segmentation structure and a skip connection mechanism until the segmentation accuracy reaches the preset requirements, and then adds the rectal tumor location information.

4. The automatic 3D fine annotation system for multimodal medical imaging staging of rectal tumors according to claim 3, characterized in that: The rectal tumor T-staging diagnostic module mainly consists of three weight matrices W Q W K W V The self-attention network and the multi-layer neural network for the T-stage task are composed of these components. The self-attention network utilizes the image medium information output by the image 3D feature extraction module to initialize the weight matrix and set parameters. The self-attention network primarily uses the following attention formula for pre-training with a large number of 3D images to capture the image medium information. After pre-training, the matrix parameters are initialized and set: Where X is an image with location markers; Q = X × W Q K T For X×W K The transpose of the matrix; V=X×W V d K Let K be the dimension of each row vector; softmax is the activation function; The T-stage task multilayer neural network consists of multiple layers of weights. The input data is multiplied by the weights of each layer to obtain the preliminary T-stage result. When training the module, the process is continuously iterated based on the difference between the generated T-stage information and the T-stage information annotated in the fine 3D labeled training dataset. The rectal tumor T-staging diagnostic module adjusts the parameters of the self-attention network and the multi-layer neural network for the T-staging task until the accuracy of the T-staging reaches the preset requirements. Then, it adds the annotation of rectal tumor T-staging diagnostic information and finally completes the 3D fine annotation of rectal tumor T-staging in medical images.

5. The automatic 3D fine annotation system for multimodal medical imaging staging of rectal tumors according to claim 1, characterized in that: The N-staging task-guided lesion lymph node 3D annotation module includes a lesion lymph node detection module and a lesion lymph node N-staging diagnosis module. The lesion lymph node detection module includes a candidate region generation network and a classification and screening mechanism; The candidate region generation network is mainly composed of image convolution and downsampling operations. First, after image convolution, the image medium information output by the image 3D feature extraction module is used to set parameters and initialize the pre-training weights. Then, after downsampling, a low-resolution image is obtained and image features are extracted. Based on the image features, several 3D candidate regions suspected to be lymph nodes are obtained. The classification and screening mechanism classifies each candidate region through a classifier and introduces a non-maximum suppression mechanism to calculate the intersection-union ratio (IUR) of the candidate region with the highest classification score and other candidate regions, and eliminates regions with excessively high IUR to reduce overlapping candidate regions. The lesion lymph node detection module compares the 3D location information of candidate regions with the finely labeled 3D lesion lymph node locations in the training set through the combined action of the candidate region generation network and the classification screening mechanism. It continuously adjusts the classification network and hyperparameters, and determines whether a region is a lesion lymph node region after the similarity between the output 3D location information of the lesion lymph node and the finely labeled 3D location of the lesion lymph node reaches a preset standard.

6. The automatic 3D fine annotation system for multimodal medical imaging staging of rectal tumors according to claim 5, characterized in that: The N-stage diagnostic module for lesion lymph nodes has the same architecture as the T-stage diagnostic module for rectal tumors; The N-staging diagnostic module for lesions in lymph nodes includes a self-attention network and a multi-layer neural network for the N-staging task. It completes the N-staging task by using the location information of the lesions in lymph nodes, the training set, and the features of the image medium. Finally, it integrates the N-staging information and the 3D location information of the lesions in lymph nodes output by the lesion lymph node detection module to complete the 3D fine annotation of the N-staging of lesions in lymph nodes in medical images.

7. The automatic 3D fine annotation system for multimodal medical imaging staging of rectal tumors according to claim 1, characterized in that: The annotation information processing module uses the 3D location information of rectal tumors and lesion lymph nodes to set the pixel value of the rectal tumor and lesion lymph node area to 255, and set the pixel value of other areas to 0 and save it to obtain the location annotation image; The diffusion generation model includes a neural network module for optimization iteration. During the generation process, starting from a known initial state, it introduces some noise or randomness through multiple iterations. The introduction of noise causes the labeled image to have small changes at each step, which gradually generates more complex and realistic samples. The forward process mainly defines the relationship between the next time step and the previous time step, obtaining the image relationship from the noise-added image to any time step t: in, The image at time t, The image represents the time t=0. Indicates noise. , express ; It is between The previous numbers were used to generate and The coefficient; The reverse process refers to the goal starting from the last... From the noisy image at time 1, the recovered image is obtained The original graph at time t is represented by the following formula for any time t. All can be considered to be from a certain This image was obtained by directly adding noise to the original image. in, Let represent a probability distribution, given from arrive Added noise That way you can get the information from the previous moment. The probability distribution; The diffusion generation model ultimately guides the generation of medical images for T and N stages with a smaller number of specified data samples through a neural network module, thereby expanding the sample size.

8. The automatic 3D fine annotation system for multimodal medical imaging staging of rectal tumors according to claim 7, characterized in that: The supervised learning module incorporates the data generated by the diffusion generation model into the fine-grained 3D annotation training dataset, expands the existing fine-grained annotation dataset, and then inputs the expanded dataset into the annotation model to train the network again, thereby improving the accuracy of the network in 3D annotation of T and N stages of medical images.

Images