A CT image whole body multi-tumor segmentation method

Abstract

The application discloses a CT image whole body multi-tumor segmentation method, first, the collected multi-site and multi-type tumor CT images are pretreated, then the pretreated images are input into a pre-trained organ segmentation model to obtain different organ segmentation results and tumor local images of different parts; then, global and local tumor segmentation models are used to extract and fuse features, organ segmentation results are used as guide information, and tumor text prompts generated by a large language model are combined to enhance the extraction of tumor specific features; finally, an accurate tumor segmentation image is obtained through a decoder gradually upsampling. The application integrates global and local features, uses organ segmentation results as guide information to form a position attention module, and enhances the adaptability of the model to different organ tumors; through a tumor text prompt driving mechanism of the large language model, specific hyperparameters for each tumor are generated, and the accuracy and robustness of the model in the multi-site and multi-type tumor segmentation task are improved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of medical computer diagnostic image processing, and particularly relates to a CT image whole-body multi-tumor segmentation method. BACKGROUND

[0002] In recent years, with the rapid development of deep learning technology, significant progress has been made in the field of medical image analysis. However, automatic segmentation of multiple tumors in whole-body CT images still faces many challenges. First, whole-body CT scans usually contain a large number of anatomical structures, and the complexity of different organs and tissues makes it difficult to accurately identify and segment tumors. Second, tumors may exhibit high heterogeneity among different patients and different parts of the same patient, which increases the difficulty of developing a universal segmentation algorithm.

[0003] Existing CT image tumor segmentation methods are mostly based on single-organ single-task design, and model design and optimization are performed for specific organs or local regions. Although these methods have achieved certain results in specific scenarios, their application range is limited and cannot adapt to the complex situation of whole-body multi-organ multi-tumor. Although some studies attempt to use multi-stage or cascade models to handle whole-body scans, these methods are often computationally complex and difficult to apply in real-time in clinical practice. Therefore, in the face of different types of tumors and diverse patient populations, existing methods often need to redesign and train new models, which increases development costs and time.

[0004] In addition, existing methods also face the following challenges when dealing with whole-body CT images: First, it is difficult to effectively balance global anatomical structure information and local tumor details; second, there is a lack of sufficient use of high-level semantic information such as tumor type and location; finally, it performs poorly in handling rare tumor types and imbalanced datasets. These limitations seriously affect the widespread application of CT image whole-body multi-tumor segmentation technology in clinical practice.

[0005] On the other hand, the acquisition and annotation of medical image data is costly, and high-quality whole-body multi-tumor datasets are relatively scarce, limiting the training effectiveness of deep learning models, especially for rare types of tumors. How to effectively utilize limited labeled data while improving the model's generalization ability for different types of tumors becomes a problem to be solved. SUMMARY

[0006] The purpose of the present application is to develop a CT image whole-body multi-tumor segmentation method that can simultaneously handle global and local information, integrate semantic knowledge, and effectively utilize limited training data.

[0007] To achieve the above purpose, the present application proposes a CT image whole-body multi-tumor segmentation method, and adopts the following technical solutions:

[0008] A CT image whole body multi-tumor segmentation method, the method comprising the following steps:

[0009] Step 1, collect CT image global calibration data sets of different parts and different types of tumors, and pre-process each collected CT image to form a sample set;

[0010] Step 2, construct a tumor segmentation model structure, including a pre-trained organ segmentation model, a global and local tumor segmentation model structure; the pre-trained organ segmentation model adopts a 3D nnU-Net model; the global and local tumor segmentation model structure adopts a 3D convolutional neural network structure, including a lightweight global feature encoder, a complex local feature encoder, a feature fusion module, a position attention module, and a decoder, an organ guiding mechanism extracts target organ position information through the position attention module, and introduces a tumor text prompt generated by a large language model to guide the extraction of specific features of the tumor locally, the feature fusion module fuses the lightweight global feature and the complex local feature, and finally outputs step by step through the decoder;

[0011] Step 3, input the pre-processed CT images into the pre-trained organ segmentation model to obtain target organ segmentation images and target tumor local images; take each target organ segmentation image as the input of the complex local feature encoder, take the corresponding pre-processed CT images as the input of the lightweight global feature encoder, take the position information extracted by the organ guiding mechanism through the position attention module to guide the model to pay attention to the tumor features in the target organ, take the text prompt generated by the large language model based on the target tumor local image to guide the model to extract the specific features of the tumor locally, and take the corresponding tumor accurate segmentation image as the output, train the constructed global and local tumor segmentation model structure, and obtain the trained global and local tumor segmentation model; the pre-trained organ segmentation model, the global and local tumor segmentation model constitute a tumor segmentation model;

[0012] Step 4, input the pre-processed CT image to be segmented into the tumor segmentation model to obtain the corresponding tumor accurate segmentation image.

[0013] Further, the CT image global calibration data set collected in step 1 includes different types of tumor data of different organs such as bone, pancreas, kidney, colon, lung, liver and esophagus, each CT image contains CT scans of the entire organ, showing the positional relationship of the tumor in the overall anatomical structure, and the size and morphology of the tumor in the data set are different; the pre-processing includes ensuring the scale and orientation of different CT images consistent by standardizing the physical distance and direction, converting CT images of different formats into a unified input format, and integrating the local and global images of the same organ and the same tumor, so that they can be simultaneously input into the pre-trained organ segmentation model.

[0014] Further, the organ segmentation model pre-trained in step 2 selects a 3D nnU-Net model, both the encoder and the decoder of which adopt a gray single channel, the channel number of the convolution kernel adopted in the down-sampling process of the encoder is 16, 32, 64, 128, 256 in turn, and the corresponding image processing size is 96x96x96, 48x48x48, 24x24x24, 12x12x12, 6x6x6 in turn, and the features extracted by the encoder are fused to the feature maps of the corresponding size of the decoder through a jump connection; the pre-trained organ segmentation model can accurately locate the target organ region, segment the target organ where the tumor exists, and obtain a target tumor local image at the same time.

[0015] Further, the global and local tumor segmentation model structure extracts the global features of the CT image through a lightweight global feature encoder and extracts the detailed features of the target organ segmentation image through a complex local feature encoder; in the encoding and decoding process, different tumor types and anatomical sites of text prompts generated based on the target tumor local image are introduced by a large language model, so as to enhance the capture of tumor specific features by the model structure; the lightweight global features and the complex local features are fused through a feature fusion module; in the decoding process, an organ guiding mechanism is introduced, and features are extracted through a position attention module to guide the model structure to focus on the tumor features of the target organ; finally, the corresponding tumor accurate segmentation image is output through the gradual up-sampling of the decoder.

[0016] Further, the large language model takes the target tumor local image as input, extracts the target tumor type and spatial position features through convolution coding, and generates a target tumor text vector; in the training process, the target tumor text vector is introduced into the encoding and decoding process of the global and local tumor segmentation model structure to guide the global and local tumor segmentation model structure to accurately capture the specific features of different tumors in the training process.

[0017] Further, the proposed organ guiding mechanism takes the segmentation image of the target organ as input, extracts features through a position attention module, assigns different attention weights to each position according to the shape, size and position of the target organ in the image, focuses on the key features related to the tumor in the target organ, and fuses the extracted features into the decoder structure, so as to enhance the attention of the model structure to the tumor features in the target organ and suppress the interference from non-target organs, thereby improving the segmentation accuracy.

[0018] The application also protects a computer device, which comprises a processor, a memory, a communication interface and a communication bus, the processor, the memory and the communication interface complete communication with each other through the communication bus, the memory is used to store at least one executable instruction, and the executable instruction makes the processor execute the corresponding operation of the CT image whole body multi-tumor segmentation method.

[0019] The application also protects a computer readable storage medium, and the computer storage medium stores at least one executable instruction, and the executable instruction enables a processor to perform operations corresponding to the CT image whole body multi-tumor segmentation method.

[0020] Compared with the prior art, the application has the following beneficial effects:

[0021] The CT image whole body multi-tumor segmentation method provided by the application firstly uses a pre-trained organ segmentation model to accurately locate an organ where a tumor is located, and extracts a corresponding tumor local region, then performs global and local feature extraction through a 3D convolutional neural network, and realizes effective fusion of global and local features; taking a target organ segmentation result as guide information not only helps the model to more accurately identify the boundary and position of a target tumor, but also enhances the adaptability of the model to tumors of different organs; in addition, the model also introduces a text prompt (prompt) driven mechanism based on a large language model to generate specific hyperparameters for different tumor sites, and further improves the accuracy and robustness of the segmentation task under complex conditions; the CT image whole body multi-tumor segmentation method provided by the application significantly improves the segmentation precision of multi-site and multi-type tumors by combining organ segmentation results and tumor local region information. BRIEF DESCRIPTION OF DRAWINGS

[0022] Figure 1 A flowchart of the CT image whole body multi-tumor segmentation method provided by the application is shown in the figure.

[0023] Figure 2 A network framework diagram of the tumor segmentation model provided by the application is shown in the figure.

[0024] Figure 3 A structure diagram of the pre-trained organ segmentation model provided by the application is shown in the figure.

[0025] Figure 4 An organ guide mechanism framework diagram provided by the application is shown in the figure. DETAILED DESCRIPTION

[0026] The technical solutions of the application will be further described in detail below in combination with the drawings and specific embodiments.

[0027] As shown in the figure, the application provides a CT image whole body multi-tumor segmentation method, which comprises the following steps: Figure 1

[0028] Step 1, collect CT image global calibration data sets of different parts and different types of tumors, and pre-process each collected CT image to form a sample set;

[0029] ​Step 2, constructing a tumor segmentation model structure, including a pre-trained organ segmentation model, a global and local tumor segmentation model structure; the pre-trained organ segmentation model adopts a 3D nnU-Net model; the global and local tumor segmentation model structure adopts a 3D convolutional neural network structure, including a lightweight global feature encoder, a complex local feature encoder, a feature fusion module, a position attention module, and a decoder, an organ guiding mechanism extracts target organ position information through the position attention module, and introduces a tumor text prompt generated by a large language model to guide the extraction of specific features of the tumor locally, the feature fusion module fuses the lightweight global feature and the complex local feature, and finally outputs step by step through the decoder after upsampling;

[0030] Step 3, inputting the preprocessed CT images into the pre-trained organ segmentation model to obtain target organ segmentation images and target tumor local images; taking each target organ segmentation image as the input of the complex local feature encoder, taking the corresponding preprocessed CT image as the input of the lightweight global feature encoder, taking the position information extracted by the organ guiding mechanism through the position attention module to guide the model to pay attention to the tumor features in the target organ, taking the text prompt generated by the large language model based on the target tumor local image to guide the model to extract the specific features of the tumor locally, and taking the corresponding tumor accurate segmentation image as the output, training the constructed global and local tumor segmentation model structure to obtain the trained global and local tumor segmentation model; the pre-trained organ segmentation model, the global and local tumor segmentation model constitute a tumor segmentation model;

[0031] Step 4, inputting the preprocessed CT image to be segmented into the tumor segmentation model to obtain the corresponding tumor accurate segmentation image.

[0032] The CT image global calibration dataset collected in step 1 includes different types of tumor data of different organs such as bone, pancreas, kidney, colon, lung, liver, and esophagus, each CT image contains CT scans of the entire organ, showing the position relationship of the tumor in the overall anatomical structure, and the size and morphology of the tumor in the dataset are various; the preprocessing includes ensuring the scale and orientation of different CT images consistent by standardizing the physical distance and direction, converting CT images of different formats into a unified input format, and integrating the local and global images of the same organ and the same tumor, so that they can be simultaneously input into the pre-trained organ segmentation model.

[0033] As Figure 3As shown, the pre-trained organ segmentation model in step 2 selects a 3D nnU-Net model, both the encoder and the decoder of which adopt a gray single channel, and the number of channels of the convolution kernel used in the downsampling process of the encoder is 16, 32, 64, 128, and 256 in turn, and the corresponding image processing is 96x96x96, 48x48x48, 24x24x24, 12x12x12, and 6x6x6 in turn, and the features extracted by the encoder are fused to the feature maps of the corresponding size of the decoder through a jump connection; the pre-trained organ segmentation model can accurately locate the target organ region, segment the target organ where the tumor exists, and obtain a target tumor local image at the same time.

[0034] As shown in Figure 2 The pre-processed CT images are input into the pre-trained organ segmentation model to obtain a target organ segmentation image and a target tumor local image; the structure of the global and local tumor segmentation model extracts global features of each CT image through a lightweight global feature encoder, extracts detailed features of the target organ segmentation image through a complex local feature encoder, and introduces text prompts about different tumor types and anatomical sites generated by a large language model in the encoding and decoding process; the lightweight global feature encoding and the complex local feature encoding are fused through a feature fusion module to fuse the lightweight global feature and the complex local feature, and the organ guiding mechanism is introduced in the decoding process to extract features through a position attention module; finally, the target tumor accurate segmentation image is output through the decoder step by step.

[0035] As shown in Figure 4 The large prophetic model takes the target tumor local image as input, extracts the type and spatial position information of the target tumor through convolution encoding, and constructs a position attention mechanism in combination with the organ segmentation result; the organ segmentation result is used to guide the attention weight, so that the model can more accurately focus on the tumor area inside the target organ and reduce the interference of the non-target organ area.

[0036] In the training process, the position attention mechanism is introduced into the decoding process of the global and local tumor segmentation model, so that the model can effectively capture the specific features of different tumor types; at the same time, the organ guiding mechanism extracts the spatial position information of the target organ region through convolution layer encoding by using the segmentation image of the target organ, and provides position guidance, so that the model structure can more accurately focus on the tumor area inside the target organ and reduce the interference of the non-target organ area.

[0037] The application also protects a computer device, which comprises a processor, a memory, a communication interface and a communication bus, the processor, the memory and the communication interface complete communication with each other through the communication bus, the memory is used to store at least one executable instruction, and the executable instruction makes the processor execute the operation corresponding to the CT image whole body multi-tumor segmentation method.

[0038] The application also protects a computer readable storage medium, and the computer storage medium stores at least one executable instruction, and the executable instruction makes the processor execute the operation corresponding to the CT image whole body multi-tumor segmentation method.

[0039] The above is only the preferred specific implementation of the present application, and is not used to limit the present application, and any modification, equivalent replacement, improvement, etc. made by any person skilled in the art according to the technical solution and the inventive concept of the present application within the technical range disclosed by the present application should be covered in the protection range of the present application. Therefore, the protection range of the present application should be subject to the protection range of the claims.

Claims

1. A method for segmenting multiple tumors throughout the body in CT images, characterized in that, The method includes the following steps: Step 1: Collect global calibration datasets of CT images of tumors in different locations and of different types, and preprocess the collected CT images to form a sample set; Step 2: Construct the tumor segmentation model structure, including a pre-trained organ segmentation model, global and local tumor segmentation model structures; the pre-trained organ segmentation model adopts a 3D nnU-Net model; the global and local tumor segmentation model structures adopt a 3D convolutional neural network structure, including a lightweight global feature encoder, a complex local feature encoder, a feature fusion module, a position attention module, and a decoder. The organ guidance mechanism extracts the target organ position information through the position attention module and introduces tumor text prompts generated by a large language model to guide the extraction of specific features of the tumor. The feature fusion module fuses the lightweight global features with the complex local features, and finally outputs them by progressively upsampling through the decoder. In step 2, the pre-trained organ segmentation model is a 3D nnU-Net model. Both its encoder and decoder use single-channel grayscale. The number of channels in the convolutional kernel used in the encoder's downsampling process are 16, 32, 64, 128, and 256, respectively. The corresponding image sizes after processing are 96×96×96, 48×48×48, 24×24×24, 12×12×12, and 6×6×6, respectively. The features extracted by the encoder are fused to the feature map of the corresponding size in the decoder through skip connections. The pre-trained organ segmentation model can accurately locate the target organ region, segment the target organ where the tumor exists, and obtain a local image of the target tumor. Step 3: Input the preprocessed CT images into the pre-trained organ segmentation model to obtain the target organ segmentation image and the target tumor local image; use the target organ segmentation image as the input to the complex local feature encoder, use the corresponding preprocessed CT images as the input to the lightweight global feature encoder, use the position information extracted by the position attention module through the organ guidance mechanism to guide the model to focus on the tumor features within the target organ, use the text prompts generated by the large language model based on the target tumor local image to guide the model to extract the specific features of the tumor locality, and use the corresponding accurate tumor segmentation image as the output to train the constructed global and local tumor segmentation model structure to obtain the trained global and local tumor segmentation models; the pre-trained organ segmentation model, global and local tumor segmentation models constitute the tumor segmentation model; The global and local tumor segmentation model structure extracts global features from CT images through a lightweight global feature encoder and extracts detailed features from the target organ segmentation image through a complex local feature encoder. During encoding and decoding, a large language model is introduced to generate text prompts based on the local image of the target tumor, indicating different tumor types and anatomical locations, enhancing the model's ability to capture tumor-specific features. A feature fusion module integrates the lightweight global features with the complex local features. During decoding, an organ guidance mechanism is introduced, using a positional attention module to extract features and guide the model to focus on the tumor features of the target organ. Finally, the decoder progressively upsamples and outputs the corresponding accurately segmented tumor image. The large language model takes a local image of the target tumor as input, extracts the target tumor type and spatial location features through convolutional encoding, and generates a target tumor text vector. Subsequently, the target tumor text vector is introduced into the encoding and decoding process of the global and local tumor segmentation model structure to guide the extraction of specific features of different tumors by the global and local tumor segmentation model structure during training. The proposed organ-guided mechanism takes the segmented image of the target organ as input and extracts features through a position attention module. Based on the shape, size and position of the target organ in the image, different attention weights are assigned to each position, and the extracted features are fused into the decoder structure. This enhances the model structure's attention to the tumor features inside the target organ and suppresses interference from non-target organs. Step 4: Using the preprocessed CT image to be segmented as input, apply the tumor segmentation model to obtain the corresponding accurate tumor segmentation image.

2. The method for segmenting multiple tumors in a whole-body CT image according to claim 1, characterized in that: The global calibration dataset of CT images collected in Step 1 includes different types of tumor data from various organs such as bone, pancreas, kidney, colon, lung, liver, and esophagus. Each CT image contains a CT scan of the entire organ, showing the positional relationship of the tumor in the overall anatomical structure. Moreover, the tumors in the dataset vary in size and shape. Preprocessing includes standardizing physical distance and orientation to ensure that the scale and orientation of different CT images are consistent, converting CT images of different formats into a unified input format, and integrating local and global images of the same organ and the same tumor so that they can be used as input to the pre-trained organ segmentation model.

3. A computer device, characterized in that: It includes a processor, a memory, a communication interface, and a communication bus. The processor, memory, and communication interface communicate with each other through the communication bus. The memory is used to store at least one executable instruction, which causes the processor to perform the operation corresponding to the whole-body multi-tumor segmentation method for CT images as described in any one of claims 1-2.

4. A computer-readable storage medium, characterized in that: The computer storage medium stores at least one executable instruction that causes the processor to perform the operation corresponding to the whole-body multi-tumor segmentation method for CT images as described in any one of claims 1-2.

Images