PET-CT nasopharyngeal carcinoma image enhancement method based on deep learning

Abstract

The invention discloses a PET-CT nasopharyngeal carcinoma image enhancement method based on deep learning, and relates to the technical field of medical images, and the method comprises the following steps: S1, obtaining a PET-CT image; s2, preprocessing the CT image, and converting the CT image from a DICOM format into a common NIFTI format; s3, image segmentation and feature extraction; s4, according to the image segmentation and feature extraction result of each slice position, constructing a deep learning data set by taking the head and neck anatomy structure as the reference feature of the CT image and combining with the PET image feature; s5, training and verifying the deep learning data set; s6, realizing image fusion and splicing based on the trained neural network model; s7, performing post-processing on the fused image; and S8, evaluating the post-processed image. According to the method, the deep learning network model is utilized, information in PET and information in CT are integrated and associated, the diagnosis accuracy of medical images is improved, and meanwhile the reading burden of doctors is relieved.

Description

Technical Field

[0001] The present invention relates to the field of medical image technology, and more specifically, to a PET-CT nasopharyngeal carcinoma image enhancement method based on deep learning. Background Art

[0002] As an advanced medical device, PET-CT combines PET imaging technology with CT precise anatomical imaging technology, and plays an important role in the diagnosis and treatment guidance of malignant tumors, heart, brain and other lesions. PET can provide biological metabolic function information, and CT can provide fine anatomical positioning function, but there are limitations when used alone. PET images cannot distinguish anatomical details, have low spatial resolution, low tissue contrast and unclear anatomical structure; CT images lack functional information and are prone to false positive diagnoses.

[0003] At present, PET images and CT images are usually simply superimposed and displayed, which can assist in diagnosis but puts a burden on doctors to read the images. With the development of artificial intelligence, deep learning has become a research hotspot in the field of medical image fusion; However, existing technologies still have problems, such as imperfect image registration algorithms, artifacts or noise that may be generated after fusion, and unsatisfactory enhancement effects in image boundaries and areas with complex grayscale value transformations. In this context, how to comprehensively associate PET and CT information through a deep learning network model to improve diagnostic accuracy and reduce the burden on doctors in reading films has become an urgent problem to be solved. In view of this, we propose a PET-CT nasopharyngeal carcinoma image enhancement method based on deep learning. Summary of the invention

[0004] The purpose of the present invention is to provide a PET-CT nasopharyngeal carcinoma image enhancement method based on deep learning. The technical problem to be solved is how to integrate and associate the information in PET and CT through a deep learning network model to improve the diagnostic accuracy of medical images while reducing the burden of doctors' reading of films.

[0005] In order to solve the above technical problems, the present invention provides the following technical solutions: a PET-CT nasopharyngeal carcinoma image enhancement method based on deep learning, comprising the following steps: S1: Obtain PET-CT images, including the patient's serial CT images, PET images and corresponding diagnostic reports, and obtain the patient's serial whole-body diagnostic images through PET-CT examination; S2: preprocessing the CT images, converting the CT images from DICOM format to the commonly used NIFTI format; S3: Image segmentation and feature extraction: according to the anatomical structure characteristics of the CT image, image segmentation is performed on each position slice to obtain the image segmentation result of the position slice, and feature extraction is performed on the PET image and the obtained segmentation result at each slice position to retain the CT image features and PET image features at each image slice position; S4: Based on the image segmentation and feature extraction results of each slice position, the head and neck anatomical structure is used as the reference feature of CT images combined with PET image features to construct a deep learning dataset; S5: Training and validation of deep learning datasets: Divide the deep learning dataset into training set and validation set, train the neural network model with the training data, use the trained network model on the validation set, and evaluate the performance of the model on the validation set; S6: Image fusion and stitching based on the trained neural network model; S7: post-processing the fused image; S8: Evaluate the post-processed image.

[0006] The present invention achieves enhancement of PET-CT nasopharyngeal carcinoma images by acquiring PET-CT images, preprocessing CT images, image segmentation and feature extraction, constructing deep learning data sets, deep learning network training and verification, image fusion and splicing, image post-processing and image evaluation. The present invention utilizes a deep learning network to reconstruct a PET image without anatomical information based on the anatomical information of the CT image, and optimizes the learning network through feature extraction and similarity comparison, thereby improving the reconstruction effect of the PET image, achieving PET image detail supplementation, and enhancing the visualization effect of image boundaries and complex grayscale change areas, making the information of medical images richer in clinical diagnosis, helping doctors to observe and judge the condition of the lesions more accurately, and significantly reducing the burden of doctors in reading films.

[0007] Preferably, in step S2, the CT image is preprocessed by: Use the NumpyArraySimpleITK class in SimpleITK to read the image data in DICOM format into a Numpy array to obtain the corresponding dicom_image, and then resample and convert it using the preset formula to convert the pixel values ​​in each direction of different coordinates on each position slice from physical units to commonly used mm.

[0008] Preferably, the preset formula includes a formula for converting physical coordinates to index coordinates and a resampling formula, which are as follows: Let the spatial position vector , the conversion formula from physical coordinates to index coordinates is: ; In the formula, , , The images are , , The origin coordinate value of the direction, , They are , The pixel spacing in the direction, yes Direction slice thickness; The resampling formula is: , where is the reconstructed image, is the original image, For output image , , The pixel spacing in the direction, is the output image size related parameter, Used to control the contrast inversion of the image. , are the minimum and maximum pixel values ​​of the original image, , are the minimum and maximum pixel values ​​of the desired output image.

[0009] Preferably, the step S4 constructs a deep learning data set, specifically comprising: S401: Using the 2DU-net network model to segment the CT image, extract the edge contour, average gray value and edge gradient of the segmented anatomical structure, and correspond them with the PET image of the slice at the position and the sequence of the slice to obtain a set of feature data for each slice in the sequence; S402: Obtain a diagnosis report corresponding to the above-mentioned feature, obtain the value of each parameter index and the corresponding examination date through the diagnosis report, correspond to each feature data, and serve as reference data on the image sequence; S403: using the multiple sets of image feature data and the multiple sets of corresponding reference data on the PET-CT image sequence as training data for the sequence, and cropping corresponding data segments from the image data by random combination to obtain multiple sets of input data pairs; S404: Add corresponding labels to the training data to assist in deep learning model training.

[0010] Preferably, in step S6, image fusion and splicing are implemented based on the trained neural network model, specifically: CT and PET images of the image are extracted to obtain a trained U-net model, and the CT image and PET image output by the U-net model of each slice are fused on the fused image sequence, and the image pixel values ​​are fused according to a preset pixel fusion formula.

[0011] Preferably, the preset pixel fusion formula includes: setting the fusion weight , first calculate the two images to be fused and The grayscale mean , where is the pixel coordinate domain of the image, Represents the total number of pixels in the domain; When satisfied and hour: ; In the formula, is the set gradient threshold, represents the gradient operator, is the fused image.

[0012] Preferably, the post-processing described in step S7 is gradient enhancement, comprising the following steps: S701: Image smoothing: using the Laplace operator to smooth the image. The Laplace operator is composed of two 5*5 matrices. The matrices are specifically: ; Assume the input image is , the smoothed image is , then the image smoothing operation can be expressed as a convolution operation: , where is the above Laplacian operator matrix, Represents the convolution operation; S702: gradient inverse transformation: after the image is smoothed, gradient inverse transformation is performed using a formula; S703: Gradient enhancement, using the gradient inverse transformation result as the input of gradient transformation to obtain an image with image gradient enhancement.

[0013] Preferably, the gradient inverse transformation in step S702 is implemented using a discrete integral form, specifically: Assume that the gradient of the known image in the gradient domain and , according to the gradient definition, the original image is transformed back: , where is an initial value of the image.

[0014] Preferably, the gradient enhancement in step S703 is implemented in the following manner: Assume that the gradient enhancement function is known , through the formula , where is the gradient enhanced image, is the result of the gradient inversion transformation, , is an enhancement factor that controls the intensity of gradient enhancement.

[0015] Preferably, the evaluation method for evaluating the post-processed image in step S8 is to use SSIM structural similarity as an evaluation index to perform image comparison evaluation, and the SSIM structural similarity calculation formula is: , where , Given the compared images , The mean pixel value of , is the corresponding pixel value variance, For a given image and Pixel value covariance, , is a constant, The output value is between -1 and 1, where 0 means no similarity and 1 means exact similarity.

[0016] Compared with the prior art, the present invention has the following beneficial effects: 1. The present invention achieves enhancement of PET-CT nasopharyngeal carcinoma images through a series of steps, including acquiring PET-CT images, CT image preprocessing, image segmentation and feature extraction, building a deep learning data set, deep learning network training and verification, image fusion and splicing, image post-processing and image evaluation. A deep learning network is used to reconstruct a PET image without anatomical information based on the anatomical information of the CT image, and the learning network is optimized by feature extraction and similarity comparison, thereby improving the reconstruction effect of the PET image, achieving PET image detail supplementation, and enhancing the visualization effect of image boundaries and complex grayscale change areas, making the information of medical images richer in clinical diagnosis, helping doctors to observe and judge the condition of the lesions more accurately, and significantly reducing the burden of doctors' reading of films.

[0017] 2. The present invention also generates an effective neural network model, which performs well in the fusion enhancement of PET-CT nasopharyngeal carcinoma images. It can automatically and accurately fuse PET and CT images, which not only improves the fusion efficiency, but also shows great application potential in medical image analysis and clinical diagnosis. It is expected to become an important tool to assist doctors in diagnosis.

[0018] 3. The present invention also realizes image fusion and splicing by establishing a neural network on the image with the help of deep learning. The deep learning algorithm can automatically learn the complex relationship between image features, thereby achieving more accurate matching and splicing in the image fusion process, greatly improving the accuracy of the algorithm and reducing diagnostic errors caused by inaccurate matching.

[0019] 4. The present invention constructs training samples with comprehensive reference to multiple key aspects such as the head and neck anatomical structure, PET image features, and corresponding diagnostic reports. This multi-dimensional consideration enables the trained model to better adapt to different image features when fusing images, effectively avoiding the problem of artifacts or noise generated after the fusion of images due to improper enhancement methods, ensuring the quality of the fused image, and providing a more reliable image basis for medical diagnosis.

[0020] 5. The present invention solves the problem in the prior art that the enhancement effect of PET images and CT images at image boundaries and in areas with complex grayscale value changes is not ideal. In these key areas, the present invention can better highlight the characteristics of the lesions, enhance image contrast, and enable doctors to observe the details of the lesions more clearly, thereby more accurately judging the condition and improving the accuracy and reliability of diagnosis. DETAILED DESCRIPTION

[0021] The present invention relates to a PET-CT nasopharyngeal carcinoma image enhancement method based on deep learning, comprising the following steps: S1: Obtain PET-CT images, including the patient's serial CT images, PET images and corresponding diagnostic reports, and obtain the patient's serial whole-body diagnostic images through PET-CT examination; In an embodiment of the present invention, the diagnosis report in step S1 includes a medical history summary, an examination date, and a series of parameter indicators, wherein the examination date is used to make the acquired CT image and PET image consistent in time, and the series of parameter indicators are used to assist in determining whether the image is normal and provide background information for subsequent quantitative analysis of the image; S2: preprocessing the CT images, converting the CT images from DICOM format to the commonly used NIFTI format; In the embodiment of the present invention, the CT image is preprocessed in step S2 to convert the CT image from the DICOM format to the commonly used NIFTI format. The specific method is: First, use the NumpyArraySimpleITK class in SimpleITK to read the DICOM format image data into the Numpy array to obtain the corresponding dicom_image. Then, resample and convert it using the preset formula to convert the pixel values ​​in each direction of different coordinates on each position slice from physical units to commonly used mm. In an embodiment of the present invention, the dicom_image includes Spacing, Origin and pixel matrix of the three-dimensional image; In an embodiment of the present invention, the preset formula includes a formula for converting physical coordinates to index coordinates and a resampling formula, which are as follows: Let the spatial position vector , the conversion formula from physical coordinates to index coordinates is: ; In the formula, , , The images are , , The origin coordinate value of the direction, , They are , The pixel spacing in the direction, yes Direction slice thickness; The resampling formula is: ; In the formula, is the reconstructed image, is the original image, For output image , , The pixel spacing in the direction, is the output image size related parameter, Used to control the contrast inversion of the image. , are the minimum and maximum pixel values ​​of the original image, , are the minimum and maximum pixel values ​​of the desired output image; S3: Image segmentation and feature extraction: According to the anatomical structure characteristics of the CT image, image segmentation is performed on each position slice to obtain the image segmentation result of the position slice; at each slice position, feature extraction is performed on the PET image and the obtained segmentation result to retain the CT image features and PET image features at each image slice position; S4: Based on the image segmentation and feature extraction results of each slice position obtained in step S3, the head and neck anatomical structure is used as a reference feature of the CT image and combined with the PET image features to construct a deep learning data set; In an embodiment of the present invention, the step S4 constructs a deep learning data set, specifically comprising: S401: Using the 2DU-net network model to segment the CT image, extract the edge contour, average gray value and edge gradient of the segmented anatomical structure, and correspond them with the PET image of the slice at the position and the sequence of the slice to obtain a set of feature data for each slice in the sequence; S402: Obtain a diagnosis report corresponding to the feature in step S401, obtain the value of each parameter index and the corresponding inspection date through the diagnosis report, correspond to each feature data, and serve as reference data on the image sequence; S403: using the multiple sets of image feature data and the multiple sets of corresponding reference data on the PET-CT image sequence as training data for the sequence, and cropping corresponding data segments from the image data by random combination to obtain multiple sets of input data pairs; S404: Add corresponding labels to the training data, where the labels are used to assist in the training of the deep learning model. For the image enhancement method of the PET-CT image sequence, the labels are input data pairs, which are the target effects of the image data combination for enhanced learning.

[0022] The training samples are constructed by comprehensively referring to multiple key aspects such as the anatomical structure of the head and neck, PET image features, and corresponding diagnostic reports. This multi-dimensional consideration enables the trained model to better adapt to different image features when fusing images, effectively avoiding the problem of artifacts or noise generated after fusion of images due to improper enhancement methods, ensuring the quality of the fused images, and providing a more reliable image basis for medical diagnosis. S5: Training and validation of deep learning datasets: Divide the deep learning dataset into training set and validation set, train the neural network model with the training data, use the trained network model on the validation set, and evaluate the performance of the model on the validation set; S6: Image fusion and stitching based on the trained neural network model; In an embodiment of the present invention, the image fusion and splicing is realized based on the trained neural network model in step S6, specifically: extracting the CT and PET images of the image to obtain the trained U-net model, performing image fusion on the CT image and PET image output by the U-net model of each slice on the fused image sequence, and fusing the image pixel values ​​according to a preset pixel fusion formula; In an embodiment of the present invention, the preset pixel fusion formula includes: setting the fusion weight , first calculate the two images to be fused and The grayscale mean ; ; In the formula, is the pixel coordinate domain of the image, Represents the total number of pixels in the domain; When satisfied and hour: ; In the formula, is the set gradient threshold, represents the gradient operator, is the fused image; By generating an effective neural network model, the model performs well in the fusion enhancement of PET-CT nasopharyngeal carcinoma images. It can automatically and accurately fuse PET and CT images, which not only improves the fusion efficiency, but also shows great application potential in medical image analysis and clinical diagnosis. It is expected to become an important tool to assist doctors in diagnosis. Image fusion and stitching are achieved by building a neural network on images with the help of deep learning. The deep learning algorithm can automatically learn the complex relationship between image features, thereby achieving more accurate matching and stitching in the image fusion process, greatly improving the accuracy of the algorithm and reducing diagnostic errors caused by inaccurate matching.

[0023] S7: post-processing the fused image; The post-processing described in step S7 is gradient enhancement, which includes the following steps: S701: image smoothing, using Laplace operator to perform image smoothing; In an embodiment of the present invention, the Laplacian operator is composed of two 5*5 matrices, which are specifically: ; Assume the input image is , the smoothed image is , then the image smoothing operation can be expressed as a convolution operation: ; In the formula, is the above Laplacian operator matrix, Represents the convolution operation; S702: gradient inverse transformation: after the image is smoothed, gradient inverse transformation is performed using a formula; In an embodiment of the present invention, the gradient inverse transformation in step S702 is implemented using a discrete integral form, specifically: Assume that the gradient of the known image in the gradient domain and , according to the gradient definition, the original image is transformed back: , where is an initial value of the image; S703: gradient enhancement, using the gradient inverse transformation result as the input of gradient transformation to obtain an image with image gradient enhancement; In the embodiment of the present invention, the gradient enhancement in step S703 is implemented in the following manner: Assume that the gradient enhancement function is known , through the formula , where is the gradient enhanced image, is the result of the gradient inversion transformation, , is an enhancement coefficient used to control the intensity of gradient enhancement; S8: Evaluate the post-processed image.

[0024] In an embodiment of the present invention, the evaluation method for evaluating the post-processed image in step S8 is to use SSIM structural similarity as an evaluation index for image comparison evaluation, and the SSIM structural similarity calculation formula is: , where , Given the compared images , The mean pixel value of , is the corresponding pixel value variance, For a given image and Pixel value covariance, , is a constant, The output value is between -1 and 1, where 0 means no similarity and 1 means exact similarity.

[0025] The present invention achieves enhancement of PET-CT nasopharyngeal carcinoma images by acquiring PET-CT images, preprocessing CT images, image segmentation and feature extraction, constructing deep learning data sets, deep learning network training and verification, image fusion and splicing, image post-processing and image evaluation. The present invention utilizes a deep learning network to reconstruct a PET image without anatomical information based on the anatomical information of the CT image, and optimizes the learning network through feature extraction and similarity comparison, thereby improving the reconstruction effect of the PET image, achieving PET image detail supplementation, and enhancing the visualization effect of image boundaries and complex grayscale change areas, making the information of medical images richer in clinical diagnosis, helping doctors to observe and judge the condition of the lesions more accurately, and significantly reducing the burden of doctors in reading films.

[0026] The embodiments of the present invention disclose preferred embodiments, but are not limited thereto. A person skilled in the art can easily understand the spirit of the present invention based on the above embodiments and make different extensions and changes. However, as long as they do not deviate from the spirit of the present invention, they are all within the protection scope of the present invention.

Claims

1. A PET-CT nasopharyngeal carcinoma image enhancement method based on deep learning, characterized in that: The following steps are involved: S1: Obtain PET-CT images, including the patient's serial CT images, PET images and corresponding diagnostic reports, and obtain the patient's serial whole-body diagnostic images through PET-CT examination; S2: preprocessing the CT images, converting the CT images from DICOM format to the commonly used NIFTI format; S3: Image segmentation and feature extraction: according to the anatomical structure characteristics of the CT image, image segmentation is performed on each position slice to obtain the image segmentation result of the position slice, and feature extraction is performed on the PET image and the obtained segmentation result at each slice position to retain the CT image features and PET image features at each image slice position; S4: Based on the image segmentation and feature extraction results of each slice position, the head and neck anatomical structure is used as the reference feature of CT images combined with PET image features to construct a deep learning dataset; S5: Training and validation of deep learning datasets: Divide the deep learning dataset into training set and validation set, train the neural network model with the training data, use the trained network model on the validation set, and evaluate the performance of the model on the validation set; S6: Image fusion and stitching based on the trained neural network model; S7: post-processing the fused image; S8: Evaluate the post-processed image.

2. The method for enhancing PET-CT nasopharyngeal carcinoma images based on deep learning according to claim 1, characterized in that: In step S2, the CT image is preprocessed, and the specific method is as follows: Use the NumpyArraySimpleITK class in SimpleITK to read the image data in DICOM format into a Numpy array to obtain the corresponding dicom_image, and then resample and convert it using the preset formula to convert the pixel values ​​in each direction of different coordinates on each position slice from physical units to commonly used mm.

3. The method for enhancing PET-CT nasopharyngeal carcinoma images based on deep learning according to claim 2, characterized in that: The preset formula includes a formula for converting physical coordinates to index coordinates and a resampling formula, which are as follows: Let the spatial position vector , the conversion formula from physical coordinates to index coordinates is: ； In the formula, , , The images are , , The origin coordinate value of the direction, , They are , The pixel spacing in the direction, yes Direction slice thickness; The resampling formula is: , where is the reconstructed image, is the original image, For output image , , The pixel spacing in the direction, is the output image size related parameter, Used to control the contrast inversion of the image. , are the minimum and maximum pixel values ​​of the original image, , are the minimum and maximum pixel values ​​of the desired output image.

4. The method for enhancing PET-CT nasopharyngeal carcinoma images based on deep learning according to claim 1, characterized in that: The step S4 constructs a deep learning data set, specifically including: S401: Using the 2DU-net network model to segment the CT image, extract the edge contour, average gray value and edge gradient of the segmented anatomical structure, and correspond them with the PET image of the slice at the position and the sequence of the slice to obtain a set of feature data for each slice in the sequence; S402: Obtain a diagnosis report corresponding to the above-mentioned feature, obtain the value of each parameter index and the corresponding examination date through the diagnosis report, correspond to each feature data, and serve as reference data on the image sequence; S403: using the multiple sets of image feature data and the multiple sets of corresponding reference data on the PET-CT image sequence as training data for the sequence, and cropping corresponding data segments from the image data by random combination to obtain multiple sets of input data pairs; S404: Add corresponding labels to the training data to assist in deep learning model training.

5. The method for enhancing PET-CT nasopharyngeal carcinoma images based on deep learning according to claim 1, characterized in that: In step S6, image fusion and splicing are achieved based on the trained neural network model, specifically: CT and PET images of the image are extracted to obtain a trained U-net model, and the CT image and PET image output by the U-net model of each slice are fused on the fused image sequence, and the image pixel values ​​are fused according to a preset pixel fusion formula.

6. The method for enhancing PET-CT nasopharyngeal carcinoma images based on deep learning according to claim 5, characterized in that: The preset pixel fusion formula includes: setting the fusion weight , first calculate the two images to be fused and The grayscale mean , where is the pixel coordinate domain of the image, Represents the total number of pixels in the domain; When satisfied and hour: ； In the formula, is the set gradient threshold, represents the gradient operator, is the fused image.

7. The method for enhancing PET-CT nasopharyngeal carcinoma images based on deep learning according to claim 1, characterized in that: The post-processing described in step S7 is gradient enhancement, comprising the following steps: S701: Image smoothing: using the Laplace operator to smooth the image. The Laplace operator is composed of two 5*5 matrices. The matrices are specifically: ； Assume the input image is , the smoothed image is , then the image smoothing operation can be expressed as a convolution operation: , where is the above Laplacian operator matrix, Represents the convolution operation; S702: gradient inverse transformation: after the image is smoothed, gradient inverse transformation is performed using a formula; S703: Gradient enhancement, using the gradient inverse transformation result as the input of gradient transformation to obtain an image with image gradient enhancement.

8. The method for enhancing PET-CT nasopharyngeal carcinoma images based on deep learning according to claim 7, characterized in that: The gradient inverse transformation in step S702 is implemented using a discrete integral form, specifically: Assume that the gradient of the known image in the gradient domain and , according to the gradient definition, the original image is transformed back: , where is an initial value of the image.

9. The method for enhancing PET-CT nasopharyngeal carcinoma images based on deep learning according to claim 8, characterized in that: The gradient enhancement in step S703 is implemented in the following way: Assume that the gradient enhancement function is known. , through the formula , where is the gradient enhanced image, is the result of the gradient inversion transformation, , is an enhancement factor that controls the intensity of gradient enhancement.

10. The method for enhancing PET-CT nasopharyngeal carcinoma images based on deep learning according to claim 1, characterized in that: The evaluation method for evaluating the post-processed image in step S8 is to use SSIM structural similarity as an evaluation index for image comparison evaluation. The SSIM structural similarity calculation formula is: , where , Given the compared images , The mean pixel value of , is the corresponding pixel value variance, For a given image and Pixel value covariance, , is a constant, The output value is between -1 and 1, where 0 means no similarity and 1 means exact similarity.

Images