PET-CT image synthesis method based on slice continuity and semantic segmentation enhancement

Abstract

The present invention discloses a PET-CT image synthesis method based on slice continuity and semantic segmentation enhancement, comprising the steps of: sampling and preprocessing PET / CT data, embedding a slice continuity Transformer module into a UNet framework, constructing a slice continuity synthesis network, and synthesizing an intermediate step CT; using whole body segmentation to perform tissue segmentation on a real CT, etc., to obtain segmentation mask information of the intermediate step CT; using UNet as a generator and FCN as a discriminator, constructing a semantic segmentation enhancement network, and inputting the intermediate step CT and the segmentation mask information of the intermediate step CT into the semantic segmentation enhancement network to obtain a synthetic CT; using a combined loss function of L1 loss function and L2 to optimize the slice continuity synthesis network in S3. The present invention can convert PET images obtained from clinical scans into synthetic CT images with rich anatomical structure information, effectively improving the diagnostic value of images.

Description

Technical Field

[0001] This invention belongs to the field of medical image processing technology, and relates to medical image synthesis technology, particularly to a PET-CT image synthesis method based on slice continuity and semantic segmentation enhancement. Background Technology

[0002] Positron emission tomography (PET) and computed tomography (CT) are two widely used imaging techniques in clinical medicine today. PET, through the injection of radioactive tracers, can visualize the metabolic activities of tissues and organs at the molecular level, playing an irreplaceable role in cancer diagnosis, cardiac assessment, and research on neurological diseases. CT scans, on the other hand, use X-rays to synthesize detailed anatomical images, clearly displaying bone, soft tissue, and vascular structures, and are widely used to detect various lesions such as trauma, infection, and tumors. Combined PET-CT imaging combines the functional information of PET with the anatomical details of CT, providing a more comprehensive means of disease assessment, and therefore plays a crucial role in clinical scenarios such as tumor detection, treatment planning, and efficacy evaluation.

[0003] While combined PET-CT imaging offers significant diagnostic advantages in clinical practice, it also introduces an additional radiation burden. Both the radioactive tracer used in PET scans and the X-rays used in CT scans expose patients to radiation. Although the radiation dose per individual scan is within acceptable limits, the cumulative effect of radiation is significant for patients requiring multiple imaging sessions, such as cancer patients who need regular follow-up during diagnosis and treatment. Long-term radiation exposure can increase the risk of radiation-related diseases, including various health problems such as cancer. Children, young patients, and pregnant women are particularly sensitive to radiation; therefore, reducing radiation dose is an urgent issue that needs to be addressed in the field of medical imaging.

[0004] Against this backdrop, a synthesis method combining PET scans with CT images has been proposed to significantly reduce radiation exposure. This method primarily utilizes the functional and metabolic information contained in PET images, employing deep learning algorithms to synthesize anatomical structures similar to those found in CT scans, thus preserving the comprehensive diagnostic advantages of PET-CT imaging. This not only reduces radiation exposure but also lowers examination costs and time, which is of great significance to both patients and the healthcare system.

[0005] However, this method faces two major challenges: the semantic differences in distribution between PET and CT, and the semantic misalignment problem. PET imaging, by detecting the distribution of radiotracers in the body, primarily reflects metabolic activities and functional changes in the human body, and therefore typically has low spatial resolution and high noise levels. In contrast, CT imaging synthesizes high-resolution cross-sectional images based on the differences in X-ray absorption by different tissues, revealing the fine anatomical structures of bones, blood vessels, and soft tissues. This difference in imaging purpose and characteristics results in a significant semantic gap in CT synthesis from PET. Secondly, since PET and CT are usually obtained through asynchronous scanning, this can lead to inconsistencies in anatomical features, further increasing the difficulty of semantic alignment of data during network training. Therefore, despite the many advantages of attenuation correction methods based on direct CT synthesis from PET, their clinical application still faces significant challenges. Summary of the Invention

[0006] Objective: This invention aims to provide a PET-CT image synthesis method based on slice continuity and semantic segmentation enhancement. Building upon the semantic conversion from PET to CT, it further addresses the semantic alignment problem inherent in existing PET-to-CT synthesis methods, improving the accuracy and clarity of synthesized CT images. This invention proposes an S2SCT-SA-Net neural network architecture, the core of which is a two-step synthesis network architecture encompassing PET to intermediate-step CT and intermediate-step CT to synthesized CT. First, the PET to intermediate-step CT step employs a slice continuity synthesis network, introducing an imaging slice continuity Transformer module. By extracting image features from single and continuous slices, the network is given a selection space for synthesizing multiple slices onto a single slice, ensuring that semantically deficient PET images can be converted into semantically richer CT images. Second, the intermediate-step CT to synthesized CT step employs a semantic segmentation-enhanced synthetic adversarial network. By segmenting the intermediate step and its tissues, organs, bones, and blood vessels, and inputting the segmentation mask information along with the intermediate CT into the synthetic adversarial network, semantic image features are enhanced, thereby improving the semantic alignment performance of the synthesized CT. Finally, to improve the feature extraction capability of the intermediate CT-to-synthetic CT synthesis network, this paper embeds a feature layer loss function into the feature layer of the synthesis network. By constraining the features of the coding layer, the imaging features of the network in the decoding layer are gradually enhanced, thereby improving the quality of the final synthesized CT image.

[0007] Technical Solution: To achieve the above objectives, this invention provides a PET-CT image synthesis method based on slice continuity and semantic segmentation enhancement, comprising the following steps:

[0008] S1: Samples were taken from patients who had received the 18F-FDG reagent using a PET / CT imaging device to obtain PET / CT image pairs.

[0009] S2: Perform truncation, bed removal, and standardization preprocessing on the PET / CT images acquired in step S1.

[0010] S3: Embed the slice continuity Transformer module into the Unet framework to construct a slice continuity synthesis network. Use the preprocessed PET / CT data from step S2 as input to synthesize the intermediate CT images.

[0011] S4: Use nn-UNet as the framework for whole-body segmentation to perform tissue segmentation on the sampled CT images from step S2, and obtain the segmentation semantic information of each tissue in the whole body.

[0012] S5: The discrete sampled CT images of various tissues throughout the body from S4 are concatenated and multiplied pixel by pixel with the intermediate CT images to obtain the segmentation semantic information of the intermediate CT images.

[0013] S6: Using UNet as the generator and FCN as the discriminator, a semantic segmentation enhancement network is constructed. The segmentation mask information of the intermediate CT images from S3 and S5 is then input into the semantic segmentation enhancement network to obtain a synthesized CT image.

[0014] S7: Use the combined L1 and L2 loss functions to constrain and optimize the slice continuity synthesis network in S3, and use the L1 loss function and feature layer loss function to constrain and optimize the semantic segmentation enhancement network in S6.

[0015] Furthermore, in step S1, asynchronous sampling is performed on patients who have been injected with 18F-FDG reagent using a PET / CT imaging scanning device to obtain PET images. and CT imaging .

[0016] Furthermore, in step S2, given that clinically acquired PET / CT data may contain outliers and that real CT images contain interfering images from the patient's bed, the acquired PET / CT data is truncated and de-bed processed to remove these outliers. By value Truncation, CT data By value The images were truncated, and the bed image was manually removed from the images. The truncated and bed-removed PET / CT data were then normalized. Within the value range, preparation is made for further training of the synthetic model.

[0017] Furthermore, in step S3, the synthesis network embeds a slice continuity Transformer into the Unet generator to complete the synthesis of the input PET image into the intermediate CT image. The task is as follows: First, select S consecutive PET slices as a multi-slice, and select the first PET slice from the S slices as a single slice. Then, input the single slice and the multi-slice into a slice continuity Transformer for feature extraction. Finally, sum the extracted features using tensors to obtain the feature set. Then, a UNet network with four upsampling and downsampling layers is used as the synthesis framework, and a slice continuity Transformer module is embedded between the upsampling and downsampling layers to form a slice continuity synthesis model. The slice continuity Transformer module consists of four parts: a linear mapping layer, two attention mechanisms (slice attention and self attention within slices), and an MLP. For a single-output PET slice, the slice is first split into small patches and the feature tensor is obtained through the linear mapping layer. Then, calculate separately. Corresponding query matrix Key matrix Sum matrix Information is used to compute patch correlation features between individual PETs using the Self Attention mechanism within slices. For a given S consecutive PET slices surrounding a single PET, the S PET slices are first concatenated and segmented using tensors, and then feature tensors are obtained through linear mapping layer operations. Then calculate separately corresponding , and The matrix information is used, and the feature information of S slices is integrated using the slice attention mechanism to obtain the feature information of consecutive slices. Then, the single slice features and continuous slice features Adding them together gives Output features: Finally, using the Unet synthesis framework to extract deep and shallow features from PET / CT data, as well as features extracted by the slice Transformer. Together, they use skip link layers and upsampling operations to output intermediate CT images. .

[0018] Furthermore, in step S4, a CT whole-body segmenter is used to process the sampled CT data. The process involves skeletal and organ segmentation and merging to obtain semantic information for whole-body segmentation from sampled CT scans. First, the nn-Unet network is used as the basic framework for the sampled CT segmentation task and pre-trained on the TotalSegmentator public dataset. Then, the pre-trained model is loaded and processed using sampled CT data. Segmentation is performed to obtain the whole-body segmentation semantic information of the sampled CT scan.

[0019] Furthermore, in step S5, the whole-body segmentation semantics of the discrete sampled CT images from S4 are merged, that is, the segmented regions such as the ventricles, bones, organs, and blood vessels are merged to form a complete whole-body segmentation mask, and the whole-body segmentation mask information of the sampled CT images is combined with the CT images from the intermediate steps. Pixel multiplication is performed to obtain the segmentation mask information of the intermediate step CT.

[0020] Further, in step S6, a CT semantic segmentation enhancement network is constructed, and the segmentation mask information of the intermediate CT images from S3 and S5 are jointly input into the semantic segmentation enhancement network to obtain a synthesized CT image. First, Unet is used as the generator of the semantic segmentation enhancement network to capture multi-scale features of the CT image, and the light and dark features are fused to obtain the features of the synthesized CT image. In addition, a three-layer FCN network is used as a discriminator, and the Leaky ReLU activation function is used to distinguish the synthesized CT image from the sampled image, and the discrimination result is output to... Within the interval. Finally, the generator and discriminator networks are combined, and the intermediate step CT and its segmentation mask are input into the combined semantic segmentation enhancement network to output a synthesized CT image. .

[0021] Furthermore, in step S7, a combined loss function of L1 and L2 is used. To constrain and optimize the slice continuity synthesis network in S3, the L1 loss function and the feature layer loss function are used to constrain and optimize the semantic segmentation enhancement network in S6, as shown below:

[0022] ,

[0023] ,

[0024] ,

[0025] in, and These are all hyperparameters of the loss function. To enhance the feature layer loss of the semantic segmentation network, The first part represents the semantic segmentation enhancement network. One coding layer, Indicates the first CT features extracted from each coding layer, This represents the hyperparameter value assigned to the loss of each coding layer. , and These represent the semantic segmentation network's first and second generations, respectively. The number of channels, length, and width values ​​of the layer features.

[0026] Beneficial effects: Compared with the prior art, the present invention has the following advantages:

[0027] 1. This invention fully utilizes the performance of deep synthesis networks in extracting PET image features and synthesizing CT images. By enhancing slice continuity and segmentation semantic information, a PET-CT synthesis network is constructed, proposing a PET-CT image synthesis method based on slice continuity and semantic segmentation enhancement. This reduces the need for CT scans by using a synthesis algorithm to synthesize CT images from only PET scans of the patient, thereby significantly reducing the patient's overall radiation exposure and broadening the clinical applicability of PET scanning technology.

[0028] 2. This invention embeds a slice continuity Transformer between the upsampling and downsampling layers of the Unet generator in the PET-to-intermediate CT synthesis network, enabling the network to complete the semantic translation task from PET to intermediate CT. Specifically, the slice continuity Transformer module used in this invention employs two attention mechanisms: SelfAttention for a single slice and Slice Attention for multiple slices. SelfAttention performs matrix transformations on the patches within a single PET slice to obtain multi-head attention. , and Matrix information, and calculation and The correlation features and projected to The matrix captures features within a single PET slice. Slice Attention, on the other hand, integrates and cropps information from multiple PET slices to obtain a multi-head attention matrix. , and The QK matrix is ​​then projected onto the V matrix to capture the correlation features between multiple slice PETs. ​​Finally, this invention performs element-wise addition of the features extracted by the two attention mechanisms and embeds this module into the Unet network, providing the synthesis network with a... The slice selection space provides feature information within and between slices, as well as pixel-level features of varying depths from UNet, improving the quality of synthesized intermediate CTs.

[0029] 3. In the intermediate CT to synthetic CT stage, this invention constructs a semantic segmentation enhancement network, combining the intermediate CT image with its segmentation semantic information to further improve the semantic alignment performance of the synthetic CT image. First, this invention utilizes a whole-body segmenter, nn-UNet, to perform tissue segmentation on the intermediate CT image, obtaining detailed whole-body tissue segmentation semantic information. Next, the tissue regions in the segmentation results are merged to form a whole-body segmentation semantic mask, which is then element-wise multiplied with the intermediate CT image to synthesize a sampled CT image with the segmentation mask. Subsequently, the semantic segmentation enhancement network takes the intermediate CT image and its segmentation mask as input, using UNet as a generator to capture pixel-level features of the intermediate CT image and its segmentation mask. The generator synthesizes the synthetic CT image by combining these features. FCN acts as a discriminator, evaluating the differences between the synthetic CT image and the intermediate CT image, and continuously optimizing the generator's output through adversarial training. The discriminator's feedback helps improve the detail and structural consistency of the synthetic CT image, making it closer to the distribution of a real CT image. This invention significantly enhances the semantic alignment capability and visual quality of synthetic CT images by fusing segmentation semantic information and synthetic adversarial training. This results in more accurate and richer anatomical representations in the synthetic CT images, thereby providing high-quality clinical diagnostic support.

[0030] 4. In training the semantic segmentation enhancement network, this invention introduces a feature layer loss function to optimize the performance of the synthesis network. First, the sampled CT image and the intermediate step CT image are input into the semantic segmentation enhancement network. The synthesis result is gradually adjusted by minimizing the difference between the output distributions of the two images and the sampled distribution. When the distribution difference is reduced to a sufficiently small level, further processing includes using the j-th layer feature map of the sampled CT image before the feature coding layer as the target, and constraining the feature map of the intermediate step CT image at the same layer to make it closer to the feature map of the sampled CT image. Furthermore, when calculating the feature loss function for different layers, corresponding hyperparameters are multiplied to improve the accuracy of the feature map and the weight of the layers. Finally, by using this feature layer loss function for network training, this invention obtains more accurate feature map information, thereby synthesizing a synthetic CT image with higher semantic alignment and richer details during the upsampling process.

[0031] 5. The PET-CT image synthesis model based on slice continuity and semantic segmentation enhancement proposed in this invention showed good performance on the PET / CT clinical dataset of Peking Union Medical College Hospital, which is sufficient to prove the generalization performance and clinical value of this invention. Attached Figure Description

[0032] Figure 1 This is a schematic diagram of the neural network structure for PET-synthesized CT based on the continuity of imaging slices provided by the present invention.

[0033] Figure 2 This invention provides a schematic diagram of a network feature layer loss function.

[0034] Figure 3 This is a flowchart illustrating the neural network algorithm for PET-synthesized CT based on the continuity of imaging slices provided by the present invention.

[0035] Figure 4 The image shows a comparison of algorithms used to test synthesized CT images. Detailed Implementation

[0036] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. After reading this invention, any modifications of the invention in various equivalent forms by those skilled in the art will fall within the scope defined by the appended claims.

[0037] Example: PET and CT exhibit semantic differences in distribution and semantic misalignment. PET imaging, by detecting the distribution of radioactive tracers in the body, primarily reflects metabolic activities and functional changes, thus typically exhibiting low spatial resolution and high noise levels. In contrast, CT imaging synthesizes high-resolution cross-sectional images based on differences in X-ray absorption by different tissues, revealing the fine anatomical structures of bones, blood vessels, and soft tissues. This difference in imaging purpose and characteristics results in a significant semantic gap in PET-to-CT synthesis. Given these characteristics, PET-to-CT synthesis tasks require semantic conversion from PET to CT and enhanced semantic alignment performance. A synthesis network was constructed by enhancing slice continuity and segmentation semantic information, respectively. Therefore, this invention provides an attenuation correction algorithm for direct PET-to-CT synthesis based on imaging slice continuity. Building upon the semantic conversion from PET to CT, it further addresses the semantic alignment problem remaining in existing PET-to-CT synthesis methods, improving the accuracy and clarity of synthesized CT images.

[0038] like Figure 3 The diagram shown is a flowchart of the present invention. An algorithm for directly synthesizing CT from PET based on the continuity of imaging slices includes the following steps:

[0039] S1: Samples were taken from patients who had received the 18F-FDG reagent using a PET / CT imaging device to obtain PET / CT image pairs.

[0040] S2: Perform truncation, bed removal, and standardization preprocessing on the PET / CT images acquired in step S1.

[0041] Given that clinically acquired PET / CT data may contain outliers and that real CT images may contain interfering images from the patient's bed, the acquired PET / CT data is truncated and de-bed processed to remove these interferences. By value Truncation, CT data By value The images were truncated, and the bed image was manually removed from the images. The truncated and bed-removed PET / CT data were then normalized. Within the value range, preparation is made for further training of the synthetic model.

[0042] S3: Embed the slice continuity Transformer module into the Unet framework to construct a slice continuity synthesis network. Use the preprocessed PET / CT data from step S2 as input to construct the slice continuity synthesis network to synthesize the intermediate CT images.

[0043] like Figure 1 As shown in part (a), the slice continuity synthesis network embeds a slice continuity Transformer into the Unet generator to complete the synthesis of the input PET image into the intermediate CT image. The task is as follows: First, select S consecutive PET slices as a multi-slice, and select the first PET slice from the S slices as a single slice. Then, input the single slice and multi-slice into a slice continuity Transformer for feature extraction. Finally, sum the extracted features using tensors to obtain the feature set. Then, a Unet network containing four upsampling and downsampling layers is used as the synthesis framework, and a slice continuity Transformer module is embedded in the middle of the upsampling and downsampling layers to form a slice continuity synthesis model.

[0044] The slice continuity Transformer module consists of four parts: a linear mapping layer, two attention mechanisms (SliceAttention and SelfAttention), and an MLP. For a single-output PET slice, the slice is first split into smaller patches, and the feature tensor is obtained through the linear mapping layer. Then, calculate separately. Corresponding query matrix Key matrix Sum matrix Information is used to compute patch correlation features between individual PETs using the Self Attention mechanism within slices. For a given S consecutive PET slices surrounding a single PET, the S PET slices are first concatenated and segmented using tensors, and then feature tensors are obtained through linear mapping layer operations. Then calculate separately corresponding , and The matrix information is used, and the feature information of S slices is integrated using the slice attention mechanism to obtain the feature information of consecutive slices. Then, the single slice features and continuous slice features Adding them together gives Output features: Finally, using the Unet synthesis framework to extract deep and shallow features from PET / CT data, as well as features extracted by the slice Transformer. Together, they use skip link layers and upsampling operations to output intermediate CT images. .

[0045] S4: Use nn-UNet as the framework for whole-body segmentation to perform tissue segmentation on the sampled CT images from step S2, and obtain the segmentation semantic information of each tissue in the whole body.

[0046] like Figure 1 As shown in part (b), the semantic segmentation enhancement network uses a CT whole-body segmenter to process the sampled CT data. The process involves skeletal and organ segmentation and merging to obtain semantic information for whole-body segmentation from sampled CT scans. First, the nn-Unet network is used as the basic framework for the sampled CT segmentation task and pre-trained on the TotalSegmentator public dataset. Then, the pre-trained model is loaded and processed using sampled CT data. Segmentation is performed to obtain the whole-body segmentation semantic information of the sampled CT scan. .

[0047] S5: Concatenate the discrete sampled CT tissue segmentation semantic information from S4 and multiply it pixel-wise with the intermediate step CT image to obtain the intermediate step CT segmentation semantic information. .

[0048] like Figure 1 As shown in part (b), the semantic segmentation enhancement network merges the whole-body segmentation semantics of discrete sampled CT images, that is, it merges segmentation regions such as ventricles, bones, organs, and blood vessels to form a complete whole-body segmentation mask, and then combines the whole-body segmentation mask information of the sampled CT images with the CT images from the intermediate steps. Pixel multiplication is performed to obtain the segmentation mask information of the intermediate step CT.

[0049] S6: Using UNet as the generator and FCN as the discriminator, a semantic segmentation enhancement network is constructed. The segmentation mask information of the intermediate CT images in S3 and S5 is input into the semantic segmentation enhancement network to obtain a synthesized CT image.

[0050] like Figure 1 As shown in part (b), a CT semantic segmentation enhancement network is constructed. The segmentation mask information of the intermediate CT images in S3 and S5 are jointly input into the semantic segmentation enhancement network to obtain a synthetic CT image. First, Unet is used as the generator of the semantic segmentation enhancement network to capture multi-scale features of the CT image, and the light and dark features are fused to obtain the features of the synthetic CT image. In addition, a three-layer FCN network is used as a discriminator, and the Leaky ReLU activation function is used to distinguish the synthetic CT image from the sampled image. The discrimination result is output to... Within the interval. Finally, the generator and discriminator networks are combined, and the intermediate step CT and its segmentation mask are input into the combined semantic segmentation enhancement network to output a synthesized CT image. .

[0051] S7: Use the combined L1 and L2 loss functions to constrain and optimize the slice continuity synthesis network in S3, and use the L1 loss function and feature layer loss function to constrain and optimize the semantic segmentation enhancement network in S6.

[0052] like Figure 2 As shown, the semantic segmentation enhancement network uses a feature layer loss function to constrain the accuracy of the network's feature extraction space. First, the sampled CT image and the intermediate-step CT image are input into the semantic segmentation enhancement network. The synthesized result is gradually adjusted by minimizing the difference between the output distributions of the two images and the sampled distribution. When the distribution difference is reduced to a sufficiently small level, further processing involves using the j-th layer feature map of the sampled CT image before the feature encoding layer as the target, and constraining the feature map of the intermediate-step CT image in the same layer to make it closer to the feature map of the sampled CT image. Furthermore, when calculating the feature loss function for different layers, corresponding hyperparameters are multiplied to improve the accuracy of the feature map and the weights of the layers. Finally, by training the network using this feature layer loss function, more accurate feature map information is obtained, resulting in a synthesized CT image with higher semantic alignment and richer details during the upsampling process.

[0053] To demonstrate the effectiveness of this invention, the following comparative experiments are also provided:

[0054] Specifically, this invention selected 50 pairs of PET / CT data to test the proposed neural network method for synthesizing PET CT based on the continuity of imaging slices. The dataset was sourced from Peking Union Medical College Hospital (ethics approval number HS2015016). The PET / CT data were acquired using a PoleStar m660 PET / CT scanner, with a radioactive glucose injection dose of approximately 0.1 mCi / kg and an injection duration of 180 seconds. Each PET / CT sample included 386-589 slices with a resolution of 192×192 pixels and a PET / CT voxel size of 3.147×3.147×1.8665 mm³. After the preprocessing described in step two, the PET / CT images were used for model training and testing. The Adam optimizer was used to train the model parameters with an initial learning rate of 0.00006.

[0055] In the comparative experiments, the experimental results of our method were compared with those of U-Net, CycleCT, cGAN, and ASEG methods, respectively. The comparison results are as follows: Figure 4 As shown, the areas marked with boxes represent regions where the proposed method synthesizes CT images with more complete semantic structures. It can be seen that compared to other algorithms, the neural network method for synthesizing CT images based on the continuity of imaging slices proposed in this invention can more accurately synthesize CT images with richer semantic alignment and anatomical details, making it more clinically applicable.

[0056] It should be noted that the above embodiments are not intended to limit the scope of protection of the present invention. Equivalent transformations or substitutions made based on the above technical solutions all fall within the scope of protection of the claims of the present invention.

Claims

1. A PET-CT image synthesis method based on slice continuity and semantic segmentation enhancement, characterized in that: The method comprises the following steps: S1: Use PET / CT imaging equipment to sample the patient and obtain PET / CT image pairs. S2: Preprocess the PET / CT images acquired in step S1 by truncation, bed removal and standardization. S3: Embed the slice continuity Transformer module into the Unet framework to build a slice continuity synthesis network. Use the PET / CT images preprocessed in step S2 as the input of the slice continuity synthesis network to synthesize the intermediate step CT images. S4: Use nn-UNet as the framework of the whole body segmenter to perform tissue segmentation on the sampled CT image in step S2 to obtain the segmentation semantic information of each tissue in the whole body. The sampled CT image is the CT image preprocessed in step S2. S5: splice the semantic information of the segmentation of each tissue of the discrete sampled CT in S4 and perform pixel product with the intermediate step CT image to obtain the segmentation mask information of the intermediate step CT image. S6: Use UNet as the generator and FCN as the discriminator to build a semantic segmentation enhancement network, and input the segmentation mask information of the intermediate step CT image in S3 and the intermediate step CT image in S5 into the semantic segmentation enhancement network to obtain a synthetic CT image. S7: Use the combined loss function of L1 and L2 to constrain and optimize the slice continuity synthesis network in S3, and use the L1 loss function and feature layer loss function to constrain and optimize the semantic segmentation enhancement network in S6; In step S1, asynchronous sampling is performed on the patient injected with 18F-FDG reagent by means of PET / CT imaging scanning equipment to obtain PET imaging I PET and CT imaging I CT , In step S2, since the clinically collected PET / CT data may have abnormal values ​​and there are bed interference images in the real CT image, the collected PET / CT data is truncated and bed-removed, and the PET data I PET The value is truncated to [0, 3500], and the CT data I CT The image was truncated to [-1024, 3072], and the bed image was manually removed from the image. The truncated and bed-removed PET / CT data were normalized to the range of [0, 1]. The slice continuity synthesis network in step S3 embeds the slice continuity Transformer module into the Unet generator to complete the synthesis of the input PET image to the intermediate step CT image I eCT First, we select S consecutive PET slices as multi-slices, and select the first PET in the S slices as a single slice. Then, we input the single slice and multi-slices into the slice continuity Transformer module for feature extraction, and accumulate the extracted features by tensor elements to obtain the feature F. slice2slice Then, a Unet network containing four up- and down-sampling layers is used as the synthesis framework, and a slice-continuous Transformer module is embedded between the up- and down-sampling layers to form a slice-continuous synthesis network. The slice continuity Transformer module consists of four parts: linear mapping layer, slice attention, slice attention, self attention, and MLP. For a single output PET slice, the slice is first split into small patches and the feature tensor Z is obtained through the linear mapping layer. 0 , then calculate Z 0 The corresponding query matrix Q self , the bond matrix K self Sum value matrix V self Information, and with the help of the slice attention mechanism Self Attention calculates the patch correlation feature information F between the single PET self For the continuous S PET slices around a single PET, the S PET slices are first concatenated and split, and the feature tensor is obtained through linear mapping layer operation. Then calculate separately The corresponding Q slice , K slice and V slice Matrix information, and with the help of the slice attention mechanism Slice Attention, the feature information of S slices is integrated to obtain the feature information of continuous slices F slice , then, the single slice feature F self and continuous slice feature F slice Add them together to get F slice2slice Output features. Finally, the deep and shallow layers of PET / CT data extracted by the Unet synthesis framework and the features F extracted by the slice continuity Transformer module are slice2slice , together through the skip link layer and upsampling operation, output the intermediate step CT image I eCT , In step S4, a whole body segmenter is used to segment the sampled CT image I CT The bones and tissues are segmented and merged to obtain the whole body segmentation semantic information of the sampled CT images. First, the nn-Unet network is used as the basic framework for the sampled CT image segmentation task and pre-trained on the TotalSegmentator public dataset. After that, the pre-trained model is loaded and the sampled CT images are segmented and merged. CT Perform segmentation to obtain the whole body segmentation semantic information of the sampled CT image; In step S5, the whole body segmentation semantics of the discrete sampled CT images in S4 are merged, the ventricle, bone, organ and blood vessel segmentation regions are merged to form a complete whole body segmentation mask, and the whole body segmentation mask information of the sampled CT images is merged with the CT image I in the intermediate step. eCT Perform pixel multiplication to obtain the segmentation mask information of the intermediate step CT image.

2. The PET-CT image synthesis method based on slice continuity and semantic segmentation enhancement according to claim 1, characterized in that: The step S6 constructs a CT semantic segmentation enhancement network, and inputs the segmentation mask information of the intermediate step CT image in S3 and the intermediate step CT image in S5 into the semantic segmentation enhancement network to obtain a synthetic CT image. First, Unet is used as the generator of the semantic segmentation enhancement network to capture the multi-scale features of the CT image, and the deep and shallow features are fused to obtain the synthetic CT image features. A three-layer FCN network is used as the discriminator, and the Leaky ReLU activation function is used to distinguish the synthetic CT image from the sampled image, and the discrimination result is output to the interval [0, 1]. Finally, the generator and discriminator networks are combined, and the intermediate step CT and the intermediate step CT segmentation mask are input into the combined semantic segmentation enhancement network to output the synthetic CT image I. sCT .

3. The PET-CT image synthesis method based on slice continuity and semantic segmentation enhancement according to claim 2, characterized in that: In step S7, a combined loss function of L1 loss function and L2 loss function is used. eCT To constrain and optimize the slice continuity synthesis network in S3, the L1 loss function and the feature layer loss function are used to constrain and optimize the semantic segmentation enhancement network in S6, which are expressed as: Among them, β and λ are loss function hyperparameters, loss feature is the feature layer loss of the semantic segmentation enhancement network, j represents the jth encoding layer of the semantic segmentation enhancement network, φ j (·) represents the CT features extracted by the jth coding layer, μ j represents the hyperparameter value assigned to each encoding layer loss, C j , H j and W j They represent the number of channels, length, and width of the j-th layer features of the semantic segmentation network, respectively.

Images