PET-CT nasopharyngeal carcinoma image enhancement method based on deep learning

Abstract

The present invention discloses a deep learning-based PET-CT nasopharyngeal carcinoma image enhancement method, which relates to the field of medical image technology and includes the following steps: S1: acquiring a PET-CT image; S2: preprocessing the CT image, converting the CT image from DICOM format to the commonly used NIFTI format; S3: image segmentation and feature extraction; S4: based on the image segmentation and feature extraction results of each slice position, using the head and neck anatomical structure as a reference feature of the CT image and combining it with the PET image features to construct a deep learning dataset; S5: training and validating the deep learning dataset; S6: fusion and splicing of the images based on a trained neural network model; S7: post-processing the fused image; and S8: evaluating the post-processed image. The present invention utilizes a deep learning network model to integrate and correlate information from PET and CT, improving the diagnostic accuracy of medical images while reducing the burden on doctors reading images.

Description

Technical Field

[0001] The present invention relates to the field of medical imaging technology, and more specifically, to a PET-CT nasopharyngeal carcinoma image enhancement method based on deep learning. Background Art

[0002] PET-CT, an advanced medical device that combines PET imaging technology with the precise anatomical imaging of CT, plays a vital role in the diagnosis and treatment guidance of malignant tumors, heart, brain, and other lesions. PET provides information on biological metabolism, while CT provides precise anatomical localization. However, using either method alone has limitations. PET images cannot distinguish anatomical details, have low spatial resolution, low tissue contrast, and lack clear anatomical structures. CT images, on the other hand, lack functional information and are prone to false-positive diagnoses.

[0003] Currently, PET and CT images are usually simply superimposed and displayed. Although this can assist in diagnosis, it places a burden on doctors to interpret the images. With the development of artificial intelligence, deep learning has become a research hotspot in the field of medical image fusion.

[0004] However, existing technologies still have problems, such as imperfect image registration algorithms, artifacts and noise that may be generated after fusion, and suboptimal enhancement effects at image boundaries and in areas with complex grayscale value changes. Against this backdrop, how to comprehensively correlate PET and CT information through deep learning network models to improve diagnostic accuracy and reduce the burden on doctors has become an urgent issue. To address this, we propose a deep learning-based PET-CT nasopharyngeal carcinoma image enhancement method. Summary of the Invention

[0005] The purpose of the present invention is to provide a PET-CT nasopharyngeal carcinoma image enhancement method based on deep learning. The technical problem to be solved is how to integrate and correlate the information in PET and CT through a deep learning network model to improve the diagnostic accuracy of medical images while reducing the burden on doctors to read the images.

[0006] To solve the above technical problems, the present invention provides the following technical solution: a PET-CT nasopharyngeal carcinoma image enhancement method based on deep learning, comprising the following steps:

[0007] S1: Acquire PET-CT images, including the patient's serial CT images, PET images, and corresponding diagnostic reports, and obtain a series of whole-body diagnostic images of the patient through PET-CT examination;

[0008] S2: Preprocess the CT image and convert it from DICOM format to the commonly used NIFTI format;

[0009] S3: Image segmentation and feature extraction: Based on the anatomical structural characteristics of the CT image, image segmentation is performed on each position slice to obtain the image segmentation result of the slice at that position. Feature extraction is performed on the PET image and the obtained segmentation result at each slice position, and the CT image features and PET image features at each image slice position are retained;

[0010] S4: Based on the image segmentation and feature extraction results at each slice position, a deep learning dataset is constructed using the head and neck anatomical structure as the reference features of CT images and PET image features. Specifically, the dataset includes:

[0011] S401: Perform image segmentation on the CT image using the 2DU-net network model, extract the edge contour, average grayscale value, and edge gradient of the segmented anatomical structure, and correspond them with the PET image of the slice at the location and the sequence of the slice to obtain a set of feature data for each slice in the sequence;

[0012] S402: Obtain a diagnosis report corresponding to the above-mentioned features, obtain the values of each parameter index and the corresponding examination date from the diagnosis report, and correspond to each feature data as reference data for the image sequence;

[0013] S403: Using the multiple sets of image feature data and the multiple sets of corresponding reference data on the PET-CT image sequence as training data for the sequence, cropping corresponding data segments from the image data by random combination to obtain multiple sets of input data pairs;

[0014] S404: Add corresponding labels to the training data to assist in deep learning model training;

[0015] S5: Training and validating deep learning datasets: Divide the deep learning dataset into a training set and a validation set. Train the neural network model on the training data, use the trained network model on the validation set, and evaluate the performance of the model on the validation set.

[0016] S6: Image fusion and stitching based on the trained neural network model;

[0017] S7: post-processing the fused image;

[0018] S8: Evaluate the post-processed image.

[0019] The present invention achieves enhancement of PET-CT nasopharyngeal carcinoma images through acquiring PET-CT images, CT image preprocessing, image segmentation and feature extraction, construction of deep learning data sets, deep learning network training and verification, image fusion and splicing, image post-processing and image evaluation. Utilizing a deep learning network, PET images without anatomical information are reconstructed based on the anatomical information of the CT images, and the learning network is optimized through feature extraction and similarity comparison, thereby improving the reconstruction effect of the PET images, achieving PET image detail supplementation, and enhancing the visualization effect of image boundaries and areas with complex grayscale changes, making medical images more informative in clinical diagnosis, helping doctors to observe and judge the condition of lesions more accurately, and significantly reducing the burden on doctors to read films.

[0020] Preferably, the CT image is preprocessed in step S2, specifically by:

[0021] Use the NumpyArraySimpleITK class in SimpleITK to read DICOM format image data into a Numpy array to obtain the corresponding dicom_image. Then, resample and convert it using preset formulas to convert the pixel values in each direction of different coordinates on each position slice from physical units to commonly used mm.

[0022] Preferably, the preset formula includes a formula for converting physical coordinates to index coordinates and a resampling formula, which are as follows:

[0023] Let the spatial position vector , the conversion formula from physical coordinates to index coordinates is:

[0024] ;

[0025] Where, 、 、 The images are 、 、 The origin coordinate value of the direction, 、 They are 、 The pixel spacing in the direction, yes Directional slice thickness;

[0026] The resampling formula is: , where is the reconstructed image, is the original image, For output image 、 、 The pixel spacing in the direction, is the output image size related parameter, Used to control the contrast inversion of the image. 、 are the minimum and maximum pixel values of the original image, 、 are the minimum and maximum pixel values of the desired output image.

[0027] Preferably, in step S6, image fusion and splicing are achieved based on the trained neural network model, specifically: CT and PET images of the image are extracted to obtain a trained U-net model, and the CT image and PET image output by the U-net model of each slice are fused on the fused image sequence, and the image pixel values are fused according to a preset pixel fusion formula.

[0028] Preferably, the preset pixel fusion formula includes: assuming that the two images to be fused are and :

[0029] When satisfied and hour:

[0030] ;

[0031] Where, is the set gradient threshold, represents the gradient operator, is the fused image.

[0032] Preferably, the post-processing in step S7 is gradient enhancement, comprising the following steps:

[0033] S701: Image smoothing: using the Laplace operator to smooth the image. The Laplace operator is composed of two 5*5 matrices. The matrices are specifically:

[0034] ;

[0035] Assume the input image is , the smoothed image is , then the image smoothing operation can be expressed as a convolution operation: , where is the above Laplacian operator matrix, Represents the convolution operation;

[0036] S702: Gradient inverse transformation: After the image is smoothed, gradient inverse transformation is performed using a formula;

[0037] S703: Gradient enhancement, using the gradient inverse transformation result as the input of gradient transformation to obtain an image with gradient enhancement.

[0038] Preferably, the gradient inverse transform in step S702 is implemented using a discrete integral form, specifically:

[0039] Assume that the gradient of the known image in the gradient domain and , according to the gradient definition, the original image is transformed back: , where is an initial value of the image.

[0040] Preferably, the gradient enhancement in step S703 is implemented in the following way: Assume that the gradient enhancement function is known , through the formula , where is the gradient enhanced image, is the result of the gradient inversion transformation, , Is an enhancement coefficient that controls the intensity of gradient enhancement.

[0041] Preferably, the evaluation method for evaluating the post-processed image in step S8 is to use SSIM structural similarity as an evaluation index for image comparison evaluation, and the SSIM structural similarity calculation formula is: , where 、 Given the compared images 、 The pixel mean, 、 is the corresponding pixel value variance, For a given image and Pixel value covariance, 、 is a constant, The output value is between -1 and 1, where 0 means no similarity and 1 means exact similarity.

[0042] Compared with the prior art, the present invention has the following beneficial effects:

[0043] 1. The present invention achieves enhancement of PET-CT nasopharyngeal carcinoma images through a series of steps, including acquiring PET-CT images, CT image preprocessing, image segmentation and feature extraction, constructing a deep learning dataset, deep learning network training and verification, image fusion and splicing, image post-processing, and image evaluation. Utilizing a deep learning network, PET images without anatomical information are reconstructed based on the anatomical information of the CT images. The learning network is optimized through feature extraction and similarity comparison, thereby improving the reconstruction effect of PET images, supplementing PET image details, and enhancing the visualization effect of image boundaries and areas with complex grayscale changes. This enriches the information of medical images in clinical diagnosis, helps doctors observe and judge the condition of lesions more accurately, and significantly reduces the burden on doctors to read images.

[0044] 2. The present invention also generates an effective neural network model, which performs well in the fusion enhancement of PET-CT nasopharyngeal carcinoma images. It can automatically and accurately fuse PET and CT images, which not only improves the fusion efficiency, but also shows great application potential in medical image analysis and clinical diagnosis. It is expected to become an important tool to assist doctors in diagnosis.

[0045] 3. The present invention also achieves image fusion and splicing by establishing a neural network on the image with the help of deep learning. The deep learning algorithm can automatically learn the complex relationship between image features, thereby achieving more accurate matching and splicing in the image fusion process, greatly improving the accuracy of the algorithm and reducing diagnostic errors caused by inaccurate matching.

[0046] 4. The present invention constructs training samples with comprehensive reference to multiple key aspects such as the head and neck anatomical structure, PET image features, and corresponding diagnostic reports. This multi-dimensional consideration enables the trained model to better adapt to different image features when fusing images, effectively avoiding the problem of artifacts or noise generated after the fusion of images due to improper enhancement methods, ensuring the quality of the fused image, and providing a more reliable image basis for medical diagnosis.

[0047] 5. The present invention solves the problem in the prior art that PET and CT images have unsatisfactory enhancement effects at image boundaries and in areas with complex grayscale value transformations. In these key areas, the present invention can better highlight lesion features and enhance image contrast, enabling doctors to observe lesion details more clearly, thereby more accurately judging the condition and improving the accuracy and reliability of diagnosis. DETAILED DESCRIPTION

[0048] The present invention relates to a deep learning-based PET-CT nasopharyngeal carcinoma image enhancement method, comprising the following steps:

[0049] S1: Acquire PET-CT images, including the patient's serial CT images, PET images, and corresponding diagnostic reports, and obtain a series of whole-body diagnostic images of the patient through PET-CT examination;

[0050] In an embodiment of the present invention, the diagnostic report in step S1 includes a medical history summary, an examination date, and a series of parameter indicators. The examination date is used to ensure that the acquired CT image and PET image are consistent in time. The series of parameter indicators are used to assist in determining whether the image is normal and provide background information for subsequent quantitative analysis of the image.

[0051] S2: Preprocess the CT image and convert it from DICOM format to the commonly used NIFTI format;

[0052] In the embodiment of the present invention, the CT image is preprocessed in step S2 to convert the CT image from the DICOM format to the commonly used NIFTI format. The specific method is:

[0053] First, use the NumpyArraySimpleITK class in SimpleITK to read the DICOM format image data into a Numpy array to obtain the corresponding dicom_image. Then, resample and convert the pixel values in each direction of different coordinates on each position slice from physical units to commonly used mm using the preset formula;

[0054] In an embodiment of the present invention, the dicom_image includes Spacing, Origin and pixel matrix of the three-dimensional image;

[0055] In an embodiment of the present invention, the preset formula includes a formula for converting physical coordinates to index coordinates and a resampling formula, which are as follows:

[0056] Let the spatial position vector , the conversion formula from physical coordinates to index coordinates is:

[0057] ;

[0058] Where, 、 、 The images are 、 、 The origin coordinate value of the direction, 、 They are 、 The pixel spacing in the direction, yes Directional slice thickness;

[0059] The resampling formula is: ;

[0060] Where, is the reconstructed image, is the original image, For output image 、 、 The pixel spacing in the direction, is the output image size related parameter, Used to control the contrast inversion of the image. 、 are the minimum and maximum pixel values of the original image, 、 are the minimum and maximum pixel values of the desired output image;

[0061] S3: Image segmentation and feature extraction: Based on the anatomical structural characteristics of the CT image, image segmentation is performed on each position slice to obtain the image segmentation result of the slice at that position; at each slice position, feature extraction is performed on the PET image and the obtained segmentation result, retaining the CT image features and PET image features at each image slice position;

[0062] S4: Based on the image segmentation and feature extraction results of each slice position obtained in step S3, the head and neck anatomical structure is used as the reference feature of the CT image and combined with the PET image features to construct a deep learning dataset;

[0063] In an embodiment of the present invention, step S4 constructs a deep learning dataset, specifically including:

[0064] S401: Perform image segmentation on the CT image using the 2DU-net network model, extract the edge contour, average grayscale value, and edge gradient of the segmented anatomical structure, and correspond them with the PET image of the slice at the location and the sequence of the slice to obtain a set of feature data for each slice in the sequence;

[0065] S402: Obtain a diagnosis report corresponding to the feature in step S401, obtain the value of each parameter index and the corresponding examination date from the diagnosis report, and correspond to each feature data as reference data for the image sequence;

[0066] S403: Using the multiple sets of image feature data and the multiple sets of corresponding reference data on the PET-CT image sequence as training data for the sequence, cropping corresponding data segments from the image data by random combination to obtain multiple sets of input data pairs;

[0067] S404: Add corresponding labels to the training data. The labels are used to assist in deep learning model training. For the image enhancement method of PET-CT image sequences, the labels are input data pairs and are the target effects of image data combination for enhanced learning.

[0068] The training samples are constructed by comprehensively referring to multiple key aspects such as the head and neck anatomical structure, PET image features, and corresponding diagnostic reports. This multi-dimensional consideration enables the trained model to better adapt to different image features when fusing images, effectively avoiding the problem of artifacts or noise generated after fusion due to improper enhancement methods, ensuring the quality of the fused image and providing a more reliable image basis for medical diagnosis.

[0069] S5: Training and validating deep learning datasets: Divide the deep learning dataset into a training set and a validation set. Train the neural network model on the training data, use the trained network model on the validation set, and evaluate the performance of the model on the validation set.

[0070] S6: Image fusion and stitching based on the trained neural network model;

[0071] In an embodiment of the present invention, the image fusion and splicing is implemented based on the trained neural network model in step S6, specifically: extracting the CT and PET images of the image to obtain a trained U-net model, performing image fusion on the CT image and PET image output by the U-net model of each slice on the fused image sequence, and fusing the image pixel values according to a preset pixel fusion formula;

[0072] In an embodiment of the present invention, the preset pixel fusion formula includes: setting the fusion weight , first calculate the two images to be fused and Gray mean ;

[0073] ;

[0074] Where, is the pixel coordinate domain of the image, Indicates the total number of pixels in the domain;

[0075] When satisfied and hour:

[0076] ;

[0077] Where, is the set gradient threshold, represents the gradient operator, is the fused image;

[0078] By generating an effective neural network model, the model excels in fusion enhancement of PET-CT nasopharyngeal carcinoma images. It can automatically and accurately fuse PET and CT images, improving fusion efficiency and demonstrating significant potential for application in medical image analysis and clinical diagnosis, potentially becoming an important diagnostic tool for physicians.

[0079] By using deep learning to establish a neural network on the image to achieve image fusion and stitching, the deep learning algorithm can automatically learn the complex relationship between image features, thereby achieving more accurate matching and stitching in the image fusion process, greatly improving the accuracy of the algorithm and reducing diagnostic errors caused by inaccurate matching.

[0080] S7: post-processing the fused image;

[0081] The post-processing in step S7 is gradient enhancement, which includes the following steps:

[0082] S701: Image smoothing, using Laplace operator to perform image smoothing;

[0083] In an embodiment of the present invention, the Laplacian operator is composed of two 5*5 matrices, which are specifically:

[0084] ;

[0085] Assume the input image is , the smoothed image is , then the image smoothing operation can be expressed as a convolution operation:

[0086] ;

[0087] Where, is the above Laplacian operator matrix, Represents the convolution operation;

[0088] S702: Gradient inverse transformation: After the image is smoothed, gradient inverse transformation is performed using a formula;

[0089] In an embodiment of the present invention, the gradient inverse transform in step S702 is implemented using a discrete integral form, specifically:

[0090] Assume that the gradient of the known image in the gradient domain and , according to the gradient definition, the original image is transformed back: , where is an initial value of the image;

[0091] S703: Gradient enhancement, using the gradient inverse transformation result as the input of gradient transformation to obtain an image with gradient enhancement;

[0092] In the embodiment of the present invention, the gradient enhancement in step S703 is implemented in the following manner: Assume that the gradient enhancement function is known , through the formula , where is the gradient enhanced image, is the result of the gradient inversion transformation, , is an enhancement coefficient used to control the intensity of gradient enhancement;

[0093] S8: Evaluate the post-processed image.

[0094] In an embodiment of the present invention, the evaluation method for evaluating the post-processed image in step S8 is to use SSIM structural similarity as an evaluation index for image comparison evaluation. The SSIM structural similarity calculation formula is: , where 、 Given the compared images 、 The pixel mean, 、 is the corresponding pixel value variance, For a given image and Pixel value covariance, 、 is a constant, The output value is between -1 and 1, where 0 means no similarity and 1 means exact similarity.

[0095] The present invention achieves enhancement of PET-CT nasopharyngeal carcinoma images through acquiring PET-CT images, CT image preprocessing, image segmentation and feature extraction, construction of deep learning data sets, deep learning network training and verification, image fusion and splicing, image post-processing and image evaluation. Utilizing a deep learning network, PET images without anatomical information are reconstructed based on the anatomical information of the CT images, and the learning network is optimized through feature extraction and similarity comparison, thereby improving the reconstruction effect of the PET images, achieving PET image detail supplementation, and enhancing the visualization effect of image boundaries and areas with complex grayscale changes, making medical images more informative in clinical diagnosis, helping doctors to observe and judge the condition of lesions more accurately, and significantly reducing the burden on doctors to read films.

[0096] The embodiments disclosed in the present invention are preferred embodiments, but are not limited to them. Ordinary technicians in this field can easily understand the spirit of the present invention based on the above embodiments and make different extensions and changes. As long as they do not deviate from the spirit of the present invention, they are all within the scope of protection of the present invention.

Claims

1. A deep learning-based PET-CT nasopharyngeal carcinoma image enhancement method, characterized in that: The following steps are involved: S1: Acquire PET-CT images, including the patient's serial CT images, PET images, and corresponding diagnostic reports, and obtain a series of whole-body diagnostic images of the patient through PET-CT examination; S2: Preprocess the CT image and convert it from DICOM format to the commonly used NIFTI format; S3: Image segmentation and feature extraction: Based on the anatomical structural characteristics of the CT image, image segmentation is performed on each position slice to obtain the image segmentation result of the slice at that position. Feature extraction is performed on the PET image and the obtained segmentation result at each slice position, and the CT image features and PET image features at each image slice position are retained; S4: Based on the image segmentation and feature extraction results at each slice position, the head and neck anatomical structure is used as the reference feature of the CT image and combined with the PET image features to construct a deep learning dataset, specifically including: S401: Perform image segmentation on the CT image using the 2DU-net network model, extract the edge contour, average grayscale value, and edge gradient of the segmented anatomical structure, and correspond them with the PET image of the slice at the location and the sequence of the slice to obtain a set of feature data for each slice in the sequence; S402: Obtain a diagnosis report corresponding to the above-mentioned features, obtain the values of each parameter index and the corresponding examination date from the diagnosis report, and correspond to each feature data as reference data for the image sequence; S403: Using the multiple sets of image feature data and the multiple sets of corresponding reference data on the PET-CT image sequence as training data for the sequence, cropping corresponding data segments from the image data by random combination to obtain multiple sets of input data pairs; S404: Add corresponding labels to the training data to assist in deep learning model training; S5: Training and validating deep learning datasets: Divide the deep learning dataset into a training set and a validation set. Train the neural network model on the training data, use the trained network model on the validation set, and evaluate the performance of the model on the validation set. S6: Image fusion and stitching based on the trained neural network model; S7: post-processing the fused image; S8: Evaluate the post-processed image.

2. The method for enhancing PET-CT nasopharyngeal carcinoma images based on deep learning according to claim 1, characterized in that: In step S2, the CT image is preprocessed, and the specific method is as follows: Use the NumpyArraySimpleITK class in SimpleITK to read DICOM format image data into a Numpy array to obtain the corresponding dicom_image. Then, resample and convert it using preset formulas to convert the pixel values in each direction of different coordinates on each position slice from physical units to commonly used mm.

3. The method for enhancing PET-CT nasopharyngeal carcinoma images based on deep learning according to claim 2, characterized in that: The preset formula includes a formula for converting physical coordinates to index coordinates and a resampling formula, which are as follows: Let the spatial position vector , the conversion formula from physical coordinates to index coordinates is: ； Where, 、 、 The images are 、 、 The origin coordinate value of the direction, 、 They are 、 The pixel spacing in the direction, yes Directional slice thickness; The resampling formula is: , where is the reconstructed image, is the original image, For output image 、 、 The pixel spacing in the direction, is the output image size related parameter, Used to control the contrast inversion of the image. 、 are the minimum and maximum pixel values of the original image, 、 are the minimum and maximum pixel values of the desired output image.

4. The method for enhancing PET-CT nasopharyngeal carcinoma images based on deep learning according to claim 1, characterized in that: In step S6, image fusion and splicing are achieved based on the trained neural network model, specifically: CT and PET images of the image are extracted to obtain a trained U-net model, and the CT image and PET image output by the U-net model of each slice are fused on the fused image sequence, and the image pixel values are fused according to a preset pixel fusion formula.

5. The method for enhancing PET-CT nasopharyngeal carcinoma images based on deep learning according to claim 4, characterized in that: The preset pixel fusion formula includes: assuming that the two images to be fused are and : When satisfied and hour: ； Where, is the set gradient threshold, represents the gradient operator, is the fused image.

6. The method for enhancing PET-CT nasopharyngeal carcinoma images based on deep learning according to claim 1, characterized in that: The post-processing described in step S7 is gradient enhancement, which includes the following steps: S701: Image smoothing: using the Laplace operator to smooth the image. The Laplace operator is composed of two 5*5 matrices. The matrices are specifically: ； Assume the input image is , the smoothed image is , then the image smoothing operation can be expressed as a convolution operation: , where is the above Laplacian operator matrix, Represents the convolution operation; S702: Gradient inverse transformation: After the image is smoothed, gradient inverse transformation is performed using a formula; S703: Gradient enhancement, using the gradient inverse transformation result as the input of gradient transformation to obtain an image with gradient enhancement.

7. The method for enhancing PET-CT nasopharyngeal carcinoma images based on deep learning according to claim 6, characterized in that: The gradient inverse transformation in step S702 is implemented using a discrete integral form, specifically: Assume that the gradient of the known image in the gradient domain and , according to the gradient definition, the original image is transformed back: , where is an initial value of the image.

8. The method for enhancing PET-CT nasopharyngeal carcinoma images based on deep learning according to claim 7, characterized in that: The gradient enhancement in step S703 is achieved by: assuming that the gradient enhancement function , through the formula , where is the gradient enhanced image, is the result of the gradient inversion transformation, , Is an enhancement coefficient that controls the intensity of gradient enhancement.

9. The method for enhancing PET-CT nasopharyngeal carcinoma images based on deep learning according to claim 1, characterized in that: The evaluation method for evaluating the post-processed image in step S8 is to use SSIM structural similarity as an evaluation index for image comparison evaluation. The SSIM structural similarity calculation formula is: , where 、 Given the compared images 、 The mean pixel value of 、 is the corresponding pixel value variance, For a given image and Pixel value covariance, 、 is a constant, The output value is between -1 and 1, where 0 means no similarity and 1 means exact similarity.

Images