Virtual enhanced CT generation method based on visual self-attention neural network

Abstract

The invention relates to the technical field of medical image processing, in particular to a virtual enhanced CT (computed tomography) generation method based on a visual self-attention neural network, which comprises the following steps: collecting MRI (magnetic resonance imaging) and CT image data of more than 200 patients with nasopharyngeal carcinoma, performing standardization, resampling, cutting and normalization preprocessing, performing deformation registration by using a CycleFCNs neural network, and obtaining an MRI (magnetic resonance imaging) and CT image data of more than 200 patients with nasopharyngeal carcinoma; then synthesizing a virtual enhanced CT by using a VIT neural network model based on a visual self-attention mechanism, and finally evaluating the similarity between the virtual enhanced CT and a real CT by using a statistical method so as to optimize the model. The advanced neural network technology is utilized, multi-mode MRI information is fully fused, the registration precision is effectively improved, the generated virtual enhanced CT is high in accuracy, a reliable basis can be provided for radiotherapy, the clinical application value is improved, accurate radiotherapy of nasopharynx cancer is assisted, and the method has important significance and wide application prospects in the field of medical image processing.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image processing, and in particular to a virtual enhanced CT generation method based on a visual self-attention neural network. Background Art

[0002] During the radiotherapy of nasopharyngeal carcinoma, as the treatment progresses, the anatomical structure of the patient's tumor and normal tissues and organs will change, which requires adjustment to the treatment plan to ensure the accuracy of radiotherapy, while effectively killing the tumor while minimizing the exposure to normal tissues and organs, thereby reducing the risk of radiotherapy complications and improving the patient's treatment effect and quality of life.

[0003] At present, MRI-based online adaptive radiotherapy technology is an emerging radiotherapy technology. It has the advantages of no radiation, high soft tissue resolution, and the ability to monitor anatomical changes in real time, which has significant advantages over traditional radiotherapy technology. However, MRI images themselves lack electron density information. This defect makes it impossible to use them directly for dose calculation, which in turn limits the widespread application of this technology in clinical practice. Therefore, how to accurately convert MRI images into images with electron density information has become a key issue that needs to be solved in the development of this technology.

[0004] In the field of medical image processing, many studies are devoted to solving this problem, among which the synthesis technology of pseudo-CT (or virtual enhanced CT) has become a research hotspot. By using advanced technologies such as deep learning, pseudo-CT images with electron density information can be generated from MRI images, which is expected to fill the gap in MRI dose calculation. However, the existing pseudo-CT synthesis methods still face many challenges, such as the accuracy of synthesis and the registration accuracy of images of different modalities, which have not been fully solved. Therefore, a virtual enhanced CT generation method based on visual self-attention neural network is proposed to address the above problems. Summary of the invention

[0005] The purpose of the present invention is to provide a virtual enhanced CT generation method based on a visual self-attention neural network to solve the problems raised in the above background technology.

[0006] To achieve the above object, the present invention provides the following technical solutions:

[0007] A virtual enhanced CT generation method based on a visual self-attention neural network comprises the following steps:

[0008] Step S1, data collection: collect MRI and CT imaging data of nasopharyngeal carcinoma patients, and ensure that the data set contains at least 200 patients;

[0009] Step S2, data preprocessing: preprocessing the MRI and CT image data, including standardization, resampling, cropping and normalization;

[0010] Step S3, deformation registration: using the CycleFCNs neural network model to perform deformation registration on MRI and CT image data;

[0011] Step S4, virtual enhanced CT synthesis: constructing a VIT neural network model based on the visual self-attention mechanism to perform virtual enhanced CT synthesis on the registered MRI images;

[0012] Step S5, similarity evaluation and optimization: using the real CT as a reference, the similarity between the synthesized virtual enhanced CT and the real CT is evaluated by statistical methods to optimize the parameters of the neural network model.

[0013] As a preferred solution, in data preprocessing, standardization processing is used to decentralize the training image data. The standardization processing formula is: ;

[0014] in, represents the normalized image, represents the original image, is the image pixel mean, represents the standard deviation of the image data, Indicates the number of pixels in the image;

[0015] Normalization processing makes the image data distributed in the (-1,1) interval. The normalization processing formula is: ;

[0016] in, represents the normalized image, represents the original image, Representing images The maximum value of Representing images The minimum value of .

[0017] As a preferred solution, in deformable registration, the CycleFCNs neural network model uses Python3 as the computer programming language, and is built with the help of the Pytorch deep learning framework, PyCharm integrated development environment, GPU computing platform CUDA and GPU acceleration library cuDNN; the CycleFCNs neural network model uses the modality-independent neighborhood descriptor as the content loss term, L2 regularization as the regularization loss term and an additional cycle loss term for training, and uses the strategy of simultaneous training of dual fully convolutional neural networks to increase the cycle consistency of the deformable registration results, and uses Res Net Block to increase the complexity of the neural network.

[0018] As a preferred solution, the CycleFCNs neural network model consists of two deformation generation networks, which input the image pair to be registered and output the spatial deformation transformation. The specific process is: input the image pair to be registered into the deformation generation network to generate the spatial deformation transformation Acting on the floating image to get the deformed image, the deformed image is input into the network again to generate Act on the deformed image to obtain the reconstructed image, and finally calculate the target loss function.

[0019] As a preferred solution, deformable registration evaluates the registration result by calculating the DSC value and HD value of the normal tissue and organ contours between the registered images based on the normal tissue and organ contours drawn by the physician. The calculation formula of the DSC value is: ;

[0020] in, and are the contours of two images to be compared, ∩ represents the intersection, Indicates the cardinality of the set. The better the deformable registration result, the larger the DSC value and the smaller the HD value.

[0021] As a preferred solution, when constructing a VIT neural network model based on the visual self-attention mechanism, the VIT neural network is constructed based on the Transformer model. The VIT neural network includes an input layer, a Patch Embedding layer, a position encoding layer, a Transformer Encoder layer and an output layer. The input layer is responsible for receiving MRI image data. The Patch Embedding layer divides the input image into image blocks of fixed size and performs linear projection, mapping the high-dimensional image data to a low-dimensional vector space for subsequent calculations; the position encoding layer adds position information to each image block so that the network can perceive the spatial structure of the image; the Transformer Encoder layer is the core part of the model, which includes multiple self-attention sublayers and feedforward neural network sublayers for extracting image features and integrating global information; the output layer maps the processed feature vector back to the original image size to generate the final virtual enhanced CT image.

[0022] As a preferred solution, in the VIT neural network model based on the visual self-attention mechanism, the implementation of the self-attention mechanism includes:

[0023] Calculate the attention score: In the self-attention sublayer of the Transformer Encoder layer, for each image patch vector, calculate the attention score between it and other image patch vectors; the specific calculation process is to obtain the query, key, and value vectors through linear transformation of the image patch vectors, and then calculate the attention score between the query and the key through the dot product operation. The query of image patch vectors: and The image patch vector : , attention score ,in, Indicates The query vector of the image patch vector The transpose of

[0024] Attention score normalization: Normalize the calculated attention score and use the softmax function to obtain the normalized attention weight. ,in, is the total number of image blocks, is the normalized attention weight; the degree of attention of each image block vector to other image block vectors is quantified as a weight value between 1 and 1, and the sum of all weights is 1;

[0025] Weighted summation: According to the normalized attention weight, the value vector of each image block vector is weighted summed to obtain the output of the self-attention layer; that is, ,in, For the The output of the image block vector after the self-attention mechanism, For the The value vector of image patch vectors.

[0026] As a preferred solution, in the similarity evaluation and optimization, the statistical method includes calculating the mean absolute error MAE, root mean square error RMSE, mean error ME, structural similarity SSIM, peak signal-to-noise ratio PSNR and Dyss similarity coefficient DSC between the synthesized virtual enhanced CT and the real CT. The MAE calculation formula is: ;in is the number of image pixels, and The virtual enhanced CT and real CT are The gray value of a pixel, For virtual enhanced CT, It is the real CT; The calculation formula is: ; The calculation formula is ; The SSIM calculation formula is: ;in, and They are virtual enhanced CT and real CT images, and is the image mean, and is the image standard deviation, is the covariance, , , is the pixel value range, , ; The PSNR calculation formula is: ;in is the number of bits of the image, and MSE is the mean square error, that is: ; When the accuracy of virtual enhanced CT is higher, , , The smaller, , , The bigger.

[0027] As a preferred solution, similarity assessment and optimization also includes dosimetric evaluation. The dosimetric evaluation is based on the RayStation radiotherapy planning system. The radiotherapy plan is designed for the test case based on real CT, and then the plan is copied to the virtual enhanced CT for dose calculation. The dose distribution of the real CT and virtual enhanced CT is obtained respectively. The evaluation indicators include MAE, RMSE, ME and gamma pass rate. The gamma pass rate calculation formula is: ;

[0028] in, The difference in dosage is the dose tolerance, The distance difference is the distance tolerance, Number of points to meet dose and distance tolerances is the total number of points. When the virtual enhanced CT dose distribution is closer to the real CT dose distribution, the MAE, RMSE, and ME are smaller, and the gamma pass rate is greater.

[0029] It can be seen from the technical solution provided by the present invention that the virtual enhanced CT generation method based on visual self-attention neural network provided by the present invention has the following beneficial effects:

[0030] 1. The present invention adopts the independently proposed CycleFCNs neural network for deformable registration. The neural network uses a number of advanced technologies in the construction process. It uses the modality-independent neighborhood descriptor (MIND) as the content loss term, which can accurately extract image structure information and effectively judge the relationship between image voxels, so as to better constrain the deformable registration process and make the registration result more in line with the actual anatomical structure. At the same time, L2 regularization is used to prevent overfitting and ensure the generalization ability of the model. The dual full convolutional neural network simultaneous training strategy and additional cycle loss terms are used to increase the cycle consistency of the deformable registration result. With the help of Res Net Block, the complexity of the neural network is improved to enhance the ability to extract complex image features. These technical means work together to overcome the shortcomings of the low accuracy of traditional deformable registration methods, significantly improve the registration accuracy of MRI and CT images, and provide a solid foundation for the subsequent accurate synthesis of virtual enhanced CT (sCT).

[0031] 2. The self-attention mechanism of the Vision-Transformer (VIT) neural network is used to break through the limitation that traditional convolutional neural networks can only focus on local information. When processing multimodal MRI information (T1, T1WI and T2 modalities), the self-attention mechanism can automatically learn the correlation between various regions in different modal images, fully consider the global correlation of the image, and thus understand the image content more comprehensively and accurately. By effectively fusing multimodal MRI information and combining the advantages of each modality, the generated sCT image has richer details and more accurate anatomical structure information. Compared with the sCT synthesized by traditional methods, the sCT generated by the present invention is significantly reduced in terms of mean absolute error (MAE), root mean square error (RMSE), mean error (ME) and other indicators when compared with the real CT (rCT), and significantly improved in terms of structural similarity (SSIM), peak signal-to-noise ratio (PSNR), and Dyss similarity coefficient (DSC), indicating that the generated sCT is closer to the rCT in terms of image features and has higher accuracy.

[0032] 3. The method of the present invention can effectively integrate multimodal MR image information, provide multi-level and multi-contrast image features for the neural network, and further improve the accuracy and reliability of sCT generation; this makes sCT have higher application value in the clinical diagnosis, treatment plan formulation, and treatment effect evaluation of nasopharyngeal carcinoma. The entire method process is rigorous, and each link is closely linked and coordinated with each other from data collection to final result evaluation to form a complete system, which can provide comprehensive and effective technical support for the precision radiotherapy of nasopharyngeal carcinoma, promote the development of nasopharyngeal carcinoma radiotherapy technology, and has important clinical significance and broad application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 The present invention is a schematic flow chart of the steps of a virtual enhanced CT generation method based on a visual self-attention neural network. DETAILED DESCRIPTION

[0034] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0035] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0036] like Figure 1 As shown, an embodiment of the present invention provides a virtual enhanced CT generation method based on a visual self-attention neural network, comprising the following steps:

[0037] Step S1, data collection: collect MRI and CT imaging data of nasopharyngeal carcinoma patients, and ensure that the data set contains at least 200 patients; the specific operations are as follows:

[0038] Step S1-1, determine the case inclusion criteria:

[0039] Cases must be confirmed as NPC patients by pathology, which is the key to ensure the accuracy and pertinence of data; only through pathological diagnosis can it be confirmed that the patient does have NPC, providing a reliable basis for subsequent research;

[0040] Medical records and radiotherapy plan data must be complete; complete medical records include the patient's basic information, medical history, treatment process records, etc., and radiotherapy plan data covers the specific radiotherapy plan, dosage and other information; these data are crucial for a comprehensive understanding of the patient's condition, evaluation of treatment effects, and subsequent data comparison and analysis;

[0041] Patients must undergo MRI examinations in this hospital and have T1, T1WI and T2 MRI images. MRI images of different modalities can provide different information. T1 images are better for displaying anatomical structures, T1WI images have advantages in displaying certain lesions, and T2 images are more sensitive to structures such as soft tissues and fluids. Having images of these three modalities at the same time can provide richer and more comprehensive information for subsequent research.

[0042] Patients with other serious underlying diseases were excluded; other serious underlying diseases may affect the patient's physiological state and imaging performance, interfering with the accuracy and reliability of the research results; for example, certain cardiovascular diseases may cause abnormal heart function, thereby affecting blood circulation, causing changes in organ perfusion in the image, and confusing the imaging characteristics of patients with simple nasopharyngeal carcinoma;

[0043] Step S1-2, determine the case exclusion criteria:

[0044] Patients who did not undergo magnetic resonance imaging were excluded; magnetic resonance imaging is an important means of obtaining high-quality imaging data. Without this examination, the MRI images required for this study cannot be obtained, and subsequent analysis and processing cannot be performed;

[0045] Cases with severe MRI image artifacts were not included in the study; severe artifacts can lead to a decline in image quality and affect the accurate identification and analysis of tissue structures and lesions in the image; for example, motion artifacts may blur the image and make it impossible to accurately determine the boundaries and morphology of the tumor; metal artifacts may mask the lesion area or create false images, leading to erroneous diagnosis and analysis results;

[0046] Patients with incomplete MRI modalities also do not meet the requirements; as mentioned above, T1, T1WI, and T2 MRI images all have their unique roles, and the lack of any one of the modalities will make the information incomplete, affecting the accuracy and reliability of the study;

[0047] Step S1-3, collecting patient imaging data:

[0048] For NPC patients who meet the inclusion criteria, their T1, T1WI and T2 MRI images will be collected; these imaging data will serve as an important basis for the study, reflecting the structure and pathological conditions of the patient's nasopharynx and surrounding tissues;

[0049] Import the collected patient MRI images into the Treatment Planning System (TPS). TPS is a professional system used to design and evaluate radiotherapy plans. Importing MRI images into it can facilitate subsequent data processing, analysis, and association with radiotherapy plans.

[0050] Export and save patient MRI images from the TPS system in DICOM format; DICOM format is a standard format for storing and transmitting medical images. It has wide compatibility and standardization, which facilitates data exchange and sharing between different devices and software, and is also conducive to subsequent data preprocessing and input of neural network models;

[0051] In the process of data collection, the integrity and accuracy of the data must be ensured, and the imaging data of each patient must be recorded and annotated in detail, including the patient's basic information, image acquisition time, equipment parameters, etc. This information is of great significance for subsequent data management, analysis, and result interpretation. For example, images collected at different times may reflect the changing process of the patient's condition, and equipment parameters may affect the quality and characteristics of the image. These factors need to be considered comprehensively when analyzing the data.

[0052] Continue to collect cases until the data set contains at least 200 patients; a larger sample size will help improve the reliability and representativeness of the research results, better reflect the characteristics and patterns of the NPC patient population, and make the neural network model trained based on these data more generalizable and applicable to a wider range of patient populations;

[0053] Step S2, data preprocessing: preprocessing the MRI and CT image data, including standardization, resampling, cropping and normalization; the specific steps are:

[0054] Step S2-1, standardization processing:

[0055] Calculate the mean of image pixels and standard deviation ; For each MRI and CT image, traverse all the pixels of the image, add up all the pixel values ​​and divide by the number of pixels , get the image pixel mean ; Then calculate the sum of the squares of the difference between each pixel value and the mean, and divide it by the number of pixels , and finally take the square root of the result to get the standard deviation ;

[0056] The image is decentralized according to the standardized processing formula: ;

[0057] For each pixel in the image , minus the calculated pixel mean , then divided by , get the standardized pixel value ; The image data processed in this way has a mean close to 0 and a relatively stable standard deviation, which helps to improve the efficiency and accuracy of subsequent neural network training; for example, when processing a batch of MRI images, after standardization, the image data of different patients are more comparable in numerical terms, and the neural network can better learn the feature information in the image;

[0058] Step S2-2, resampling:

[0059] Set the target pixel spacing according to the requirements of neural network training and actual application scenarios. For example, determine the appropriate pixel spacing based on the image resolution requirements of the neural network model used in the study and the image accuracy requirements for subsequent analysis and processing. If the model has high requirements for details, a smaller pixel spacing may be required. If computing resources are limited or more attention is paid to overall trend analysis, the pixel spacing can be appropriately increased.

[0060] Use image interpolation algorithms to resample images; common interpolation algorithms include nearest neighbor interpolation, bilinear interpolation, and cubic convolution interpolation. Taking bilinear interpolation as an example, for a target pixel, select the four nearest neighbor pixels around it according to its position in the original image, and calculate the new value of the target pixel by weighted average. The pixel spacing of the resampled image reaches the set target, and the image size may change accordingly, making it more suitable for subsequent processing and analysis.

[0061] Step S2-3, cutting:

[0062] Determine a uniform image size; determine an appropriate image size based on the input requirements of the neural network model used in the study and the focus on key areas of the image; for example, for images of nasopharyngeal carcinoma patients, focus on the nasopharynx and its surrounding areas, and determine an image size that contains key information and is suitable for model processing based on the approximate location and range of the area in the image;

[0063] Crop the image; select the area to be retained from the original image and crop it to a uniform image size; during the cropping process, ensure that the retained area contains key information in the image, such as the lesion site, important anatomical structures, etc.; for example, in MRI images, ensure that the nasopharyngeal carcinoma tumor area and the surrounding tissues and organs that may be affected are fully displayed in the cropped image;

[0064] Step S2-4, normalization processing:

[0065] Computing images The maximum value of and minimum value ; Traverse all the pixels of the image and find the maximum and minimum pixel values;

[0066] Normalize the image according to the normalization formula: ;

[0067] For each pixel in the image , substitute it into the formula to get the normalized pixel value , so that the image data is distributed in the interval (-1,1); such normalization processing helps to accelerate the convergence speed of the neural network and improve the training effect; for example, when processing CT images, the CT value range of different patients may be different. After normalization, the data range of all images is unified, and the neural network can find the optimal solution faster during the learning process; in the entire data preprocessing process, the above-mentioned standardization, resampling, cropping and normalization operations should be performed on each MRI and CT image to ensure the quality and consistency of the data, and provide a good data foundation for subsequent deformation registration, virtual enhanced CT synthesis and other steps;

[0068] Step S3, deformation registration: use the CycleFCNs neural network model to perform deformation registration on MRI and CT image data; the specific steps are:

[0069] Step S3-1, CycleFCNs neural network model construction:

[0070] Environment construction: Choose Python3 as the computer programming language, and use the Pytorch deep learning framework to build the basic architecture of the neural network; use the PyCharm integrated development environment for code writing and project management, and use the GPU computing platform CUDA and the GPU acceleration library cuDNN to give full play to the parallel computing capabilities of the GPU, accelerate the training process of the neural network, and improve computing efficiency;

[0071] Loss function and training strategy setting:

[0072] The modality-independent neighborhood descriptor (MIND) is used as the content loss term. MIND extracts image structural information by comparing the local block of an image voxel with the local block of its surrounding neighborhood. This structural information can indicate the relationship between an image voxel and its surrounding voxels, and help determine whether the voxel is in a continuous region or a boundary region, thereby better constraining the deformation of the image during the deformable registration process and making the registration result more consistent with the actual anatomical structure.

[0073] L2 regularization is used as the regularization loss term; its main function is to prevent overfitting of the neural network. By penalizing the model weights, the complexity of the model is limited to ensure that the features learned by the model on the training data have a certain generalization ability and can be applied to unseen data;

[0074] An additional cyclic loss term is introduced for training, and a dual fully convolutional neural network is used for simultaneous training. This strategy can effectively increase the cyclic consistency of the deformable registration result, that is, through the forward and reverse deformation registration process, the registration result can be kept stable and accurate in both directions, reducing the registration error.

[0075] Use Res Net Block to increase the complexity of the neural network. Res Net Block introduces residual connections, which enables the network to better learn the deep features of the image during training, enhances the model's ability to handle complex image structures and deformations, and helps improve the accuracy of registration.

[0076] Step S3-2, deformation registration process:

[0077] Input the image pair to be registered: Input the preprocessed MRI and CT image data as the image pair to be registered into the CycleFCNs neural network model; ensure that the image pair has a certain initial consistency in spatial position and direction so that the neural network can perform subsequent precise registration operations;

[0078] Generate spatial deformation transformation Deformable image: The deformation generation network in the model calculates and generates the spatial deformation transformation based on the feature information of the input image pair to be registered ; This spatial deformation transformation Describes how to map pixels in a floating image (such as an MRI image) to the spatial locations of a reference image (such as a CT image); Acts on floating images by treating each pixel in the floating image according to The defined mapping relationship is repositioned to obtain a deformed image; in this process, the model will continuously adjust the spatial deformation transformation according to the pre-set loss function (including content loss, regularization loss, and cycle loss, etc.) , so that the deformed image is closer to the reference image in structure and features;

[0079] generate And reconstructed image: The deformed image is input into the deformation generation network again, and the network generates another spatial deformation transformation ; The role of is to map the deformed image back to the spatial position of the original floating image as closely as possible, thereby obtaining a reconstructed image; this step further strengthens the model's understanding and learning ability of image deformation, and verifies and optimizes the previously generated spatial deformation transformation through reverse deformation operations. accuracy;

[0080] Calculate the target loss function: Calculate the value of the target loss function based on the original image pair to be registered, the generated deformed image and the reconstructed image; the target loss function comprehensively considers multiple factors such as content loss (measuring the structural similarity between the deformed image and the reference image through methods such as MIND), regularization loss (controlling model complexity) and cycle loss (ensuring the consistency of forward and reverse deformation registration); by calculating the target loss function, evaluate the spatial deformation transformation generated by the current model and Then, the optimization algorithm (such as stochastic gradient descent, etc.) is used to back-propagate and update the parameters of the neural network according to the gradient information of the loss function, so as to gradually reduce the value of the target loss function and continuously optimize the spatial deformation transformation. and , improve the accuracy of deformation registration; during the training process, repeat the above steps of input, generation, calculation and update until the model converges, that is, the value of the target loss function no longer decreases significantly, and the optimal spatial deformation transformation is obtained. , complete the deformation registration process of MRI and CT image data;

[0081] Step S3-3, evaluation of registration results:

[0082] Physician outline and data preparation: Invite experienced professional physicians to carefully outline the contours of normal tissues and organs in MRI and CT images; based on their professional knowledge and clinical experience, the physicians accurately depict the boundaries of important organs and tissues such as the nasopharynx, skull base, and cervical lymph nodes; the physicians’ outline data is accurately matched and integrated with the registered MRI and CT images to ensure the accuracy of the data correspondence and prepare for subsequent evaluation and calculation;

[0083] DSC value calculation: The DSC value of the normal tissue and organ contour between the registered images is calculated according to the following formula: ;

[0084] in, and are the contours of the same tissue organ in the MRI image and CT image respectively, ∩ represents the intersection, Indicates the cardinality of a set; for example, for the nasopharyngeal contour, the area of ​​the intersection of the MRI and CT images is calculated and compared with the total area of ​​the two contours; the DSC value ranges from 0 to 1, and the larger the value, the higher the overlap of the two contours, that is, the better the deformable registration result; if the DSC value is close to 1, it means that the contours of the tissue and organ in the MRI and CT images are very close in spatial position, and the registration effect is ideal; on the contrary, if the DSC value is low, it indicates that there is a large error in the registration, which requires further analysis and improvement;

[0085] HD value calculation: The surface distance between the contours of organs in different images is calculated through a specific algorithm to obtain the HD value; the HD value measures the average distance between corresponding points on the surface of the contour of the same organ in MRI and CT images; a smaller HD value means a higher spatial consistency of the contours of the organ in the two images, that is, a more accurate registration result; for example, when evaluating the registration of the skull base structure, the smaller the HD value, the better the fit of the skull base contours in the MRI and CT images, and the more accurately the anatomical structure of the skull base can be reflected, which is of great significance for subsequent applications such as virtual enhanced CT synthesis and radiotherapy planning;

[0086] Comprehensive evaluation: The registration results are comprehensively evaluated by combining the DSC value and the HD value. The larger the DSC value and the smaller the HD value, the more ideal the deformable registration result is, that is, the higher the consistency of the anatomical structure between the MRI and CT images. However, in practical applications, it may be necessary to set reasonable thresholds or weights for the DSC value and the HD value according to specific clinical needs and research purposes. For example, for some situations where the accuracy of organ contours is extremely high, more emphasis may be placed on the performance of the DSC value. For some applications that need to focus on the spatial consistency of the overall structure, the impact of the HD value may be more critical. If the registration result is not ideal, that is, the DSC value is low or the HD value is large, it may be necessary to recheck the data quality, adjust the parameters or structure of the neural network model, or improve the training strategy to improve the accuracy and reliability of the deformable registration and provide more accurate basic data for subsequent virtual enhanced CT synthesis.

[0087] Step S4, virtual enhanced CT synthesis: constructing a VIT neural network model based on the visual self-attention mechanism to perform virtual enhanced CT synthesis on the registered MRI images, specifically including:

[0088] Step S4-1, VIT neural network model construction:

[0089] Infrastructure construction: The VIT neural network is constructed based on the Transformer model; the Transformer model has shown a strong ability to process long-distance dependencies in the field of natural language processing, and its architecture has been introduced into the visual field to process medical imaging data;

[0090] Network layer design and function definition:

[0091] The input layer is responsible for receiving T1, T1WI and T2 modality MRI image data after deformation registration by the CycleFCNs neural network; the input layer performs preliminary format conversion and preprocessing on the data to make it meet the input requirements of the subsequent network layer, ensuring that the data can flow and be processed smoothly in the network;

[0092] The Patch Embedding layer divides the input MRI image into fixed-size image patches and performs linear projection on each image patch. In this way, the high-dimensional image data is mapped to a low-dimensional vector space, reducing the complexity of the data and facilitating subsequent calculations. For example, the original two-dimensional image patch is converted into a one-dimensional vector, and the feature information of the image patch is extracted, laying the foundation for subsequent feature extraction and information fusion.

[0093] The position encoding layer adds position information to each image block. Since the spatial structure information of the original image is partially lost after the image is divided into blocks, the position encoding generates a position encoding vector through a specific function (such as sine and cosine functions) and adds it to the image block vector, so that the network can perceive the information at different positions in the image, thereby better understanding the spatial structure of the image, which is crucial for accurately synthesizing virtual enhanced CT.

[0094] The Transformer Encoder layer is the core part of the model, which is composed of multiple layers with the same structure. It contains multiple self-attention sub-layers and feed-forward neural network (FFN) sub-layers. The self-attention sub-layer uses the self-attention mechanism to calculate the correlation between image blocks, so that the model can pay attention to the global information of the image and capture the dependency between different regions. The FFN sub-layer performs nonlinear transformation on each image block vector to further extract and enhance feature information. The output of each sub-layer is connected through residual connection and layer normalization, which helps to alleviate the gradient vanishing problem and improve the training stability and performance of the network.

[0095] The output layer maps the feature vector processed by the Transformer Encoder layer back to the size and resolution of the original image, and finally generates a virtual enhanced CT (sCT) image. The output layer converts the low-dimensional feature vector into an sCT image with the same format and size as the real CT image according to the pre-defined mapping rules, ensuring that the generated sCT image can be accurately compared and analyzed with other imaging data in clinical applications.

[0096] Step S4-2, the implementation of the self-attention mechanism includes:

[0097] Calculate the attention score: In the self-attention sublayer of the Transformer Encoder layer, for each image patch vector, calculate the attention score between it and other image patch vectors; the specific calculation process is to obtain the query, key, and value vectors through linear transformation of the image patch vectors, and then calculate the attention score between the query and the key through the dot product operation. The query of image patch vectors: and The image patch vector : , attention score ,in, Indicates The query vector of the image patch vector The transpose of

[0098] Step S4-3, attention score normalization:

[0099] The calculated attention scores are normalized and the softmax function is used to obtain the normalized attention weights: ;

[0100] in, is the total number of image blocks, is the normalized attention weight; the degree of attention of each image block vector to other image block vectors is quantified as a weight value between 1 and 1, and the sum of all weights is 1;

[0101] Weighted summation: According to the normalized attention weights, the value vector of each image block vector is weighted summed to obtain the output of the self-attention layer; that is: ;

[0102] in, For the The output of the image block vector after the self-attention mechanism, For the The value vector of image block vectors;

[0103] Step S4-4, model training and optimization:

[0104] Loss function selection: Select an appropriate loss function to measure the difference between the generated sCT and the real CT (rCT), such as mean square error (MSE), structural similarity index (SSIM) loss, etc. The smaller the value of the loss function, the closer the generated sCT is to the rCT, and the better the model performance is. By minimizing the loss function, the model is guided to learn the mapping relationship from the registered MRI image to the accurate virtual enhanced CT image.

[0105] Application of optimization algorithms: Use optimization algorithms (such as stochastic gradient descent (SGD), Adam, etc.) to update the parameters of the model; during the training process, the gradient of each parameter is calculated by the back propagation algorithm based on the error calculated by the loss function, and then the optimization algorithm is used to adjust the value of the parameter based on the gradient; for example, the Adam optimization algorithm combines the concepts of adaptive learning rate and momentum, and can dynamically adjust the learning rate according to the change of gradient during the training process, so that the model converges to the optimal solution faster; by continuously iterating and optimizing parameters, the accuracy of the model generating sCT is improved;

[0106] Hyperparameter adjustment: Adjust and optimize the model's hyperparameters, including learning rate, batch size, number of iterations (epochs), number of hidden layers, number of attention heads, etc.; find a set of hyperparameter combinations that can achieve the best model performance through multiple experiments and verifications; for example, grid search or random search methods can be used to traverse different hyperparameter values ​​within a certain range, use the validation set to evaluate the model performance, and select the best hyperparameter combination as the final model setting; appropriate hyperparameters can improve the model's training efficiency and generalization ability, and avoid overfitting or underfitting;

[0107] Model evaluation and validation: During the training process, the validation set is used to evaluate the model regularly. The validation set is a part of independent data separated from the training data and does not participate in the training process. By calculating the evaluation indicators on the validation set (such as MSE, SSIM, peak signal-to-noise ratio (PSNR) and other imaging indicators and indicators related to dose calculation), the performance of the model on unseen data can be understood. If the performance of the model on the validation set no longer improves or shows a downward trend, it may indicate that the model is overfitting. At this time, corresponding measures need to be taken to improve it, such as increasing the amount of training data, using regularization techniques (such as L2 regularization, Dropout, etc.) or adjusting the model structure. Through continuous evaluation and improvement, the performance of the VIT neural network model can be improved so that it can accurately synthesize virtual enhanced CT images, providing reliable support for the clinical diagnosis and treatment of nasopharyngeal carcinoma.

[0108] Step S5, similarity evaluation and optimization: using the real CT as a reference, evaluating the similarity between the synthesized virtual enhanced CT and the real CT by statistical methods, and optimizing the parameters of the neural network model;

[0109] In the similarity evaluation and optimization, the statistical methods include calculating the mean absolute error MAE, root mean square error RMSE, mean error ME, structural similarity SSIM, peak signal-to-noise ratio PSNR and Dyss similarity coefficient DSC between the synthesized virtual enhanced CT and the real CT. The MAE calculation formula is: ;

[0110] in, is the number of image pixels, and The virtual enhanced CT and real CT are The gray value of a pixel, For virtual enhanced CT, It is the real CT; The calculation formula is: ; The calculation formula is: ; The SSIM calculation formula is: ;

[0111] in, and They are virtual enhanced CT and real CT images, and is the image mean, and is the image standard deviation, is the covariance, , , is the pixel value range, , ; The PSNR calculation formula is: ;

[0112] in, is the number of bits of the image, and MSE is the mean square error, that is: ;

[0113] The higher the accuracy of virtual enhanced CT, , , The smaller, , , The bigger;

[0114] Similarity assessment and optimization also include dosimetric evaluation. The dosimetric evaluation is based on the RayStation radiotherapy planning system. The radiotherapy plan is designed for the test case based on real CT. The plan is then copied to the virtual enhanced CT for dose calculation. The dose distribution of the real CT and virtual enhanced CT are obtained respectively. The evaluation indicators include MAE, RMSE, ME and gamma pass rate. The gamma pass rate calculation formula is: ;

[0115] in, The difference in dosage is the dose tolerance, The distance difference is the distance tolerance, Number of points to meet dose and distance tolerances is the total number of points. When the virtual enhanced CT dose distribution is closer to the real CT dose distribution, the MAE, RMSE, and ME are smaller, and the gamma pass rate is greater.

[0116] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A virtual enhanced CT generation method based on visual self-attention neural network, characterized by: The following steps are involved: Step S1, data collection: collect MRI and CT imaging data of nasopharyngeal carcinoma patients, and ensure that the data set contains at least 200 patients; Step S2, data preprocessing: preprocessing the MRI and CT image data, including standardization, resampling, cropping and normalization; Step S3, deformation registration: using the CycleFCNs neural network model to perform deformation registration on MRI and CT image data; Step S4, virtual enhanced CT synthesis: constructing a VIT neural network model based on the visual self-attention mechanism to perform virtual enhanced CT synthesis on the registered MRI images; Step S5, similarity evaluation and optimization: using the real CT as a reference, the similarity between the synthesized virtual enhanced CT and the real CT is evaluated by statistical methods to optimize the parameters of the neural network model.

2. The method for generating virtual enhanced CT based on a visual self-attention neural network according to claim 1, characterized in that: In the data preprocessing, the standardization process decentralizes the training image data. The standardization process formula is: ； in, represents the normalized image, represents the original image, is the image pixel mean, represents the standard deviation of the image data, Indicates the number of pixels in the image; Normalization processing makes the image data distributed in the (-1,1) interval. The normalization processing formula is: ； in, represents the normalized image, represents the original image, Representing images The maximum value of Representing images The minimum value of .

3. The method for generating virtual enhanced CT based on a visual self-attention neural network according to claim 1, characterized in that: In the deformable registration, the CycleFCNs neural network model uses Python3 as the computer programming language, and is built with the help of the Pytorch deep learning framework, the PyCharm integrated development environment, the GPU computing platform CUDA and the GPU acceleration library cuDNN; the CycleFCNs neural network model uses the modality-independent neighborhood descriptor as the content loss term, the L2 regularization as the regularization loss term and the additional cycle loss term for training, and uses the strategy of simultaneous training of dual fully convolutional neural networks to increase the cycle consistency of the deformable registration results, and uses the Res Net Block to increase the complexity of the neural network.

4. The method for generating virtual enhanced CT based on a visual self-attention neural network according to claim 3, characterized in that: The CycleFCNs neural network model consists of two deformation generation networks, which input the image pair to be registered and output the spatial deformation transformation. The specific process is as follows: the image pair to be registered is input into the deformation generation network to generate the spatial deformation transformation Acting on the floating image to get the deformed image, the deformed image is input into the network again to generate Act on the deformed image to obtain the reconstructed image, and finally calculate the target loss function.

5. The method for generating virtual enhanced CT based on a visual self-attention neural network according to claim 4, characterized in that: The deformable registration evaluates the registration result by calculating the DSC value and HD value of the normal tissue and organ contours between the registered images based on the normal tissue and organ contours drawn by the physician. The calculation formula of the DSC value is: ； in, and are the contours of two images to be compared, ∩ represents the intersection, Indicates the cardinality of the set. The better the deformable registration result, the larger the DSC value and the smaller the HD value.

6. The method for generating virtual enhanced CT based on a visual self-attention neural network according to claim 1, characterized in that: When constructing the VIT neural network model based on the visual self-attention mechanism, the VIT neural network is constructed based on the Transformer model. The VIT neural network includes an input layer, a Patch Embedding layer, a position encoding layer, a Transformer Encoder layer and an output layer. The input layer is responsible for receiving MRI image data. The Patch Embedding layer divides the input image into image blocks of fixed size and performs linear projection, mapping the high-dimensional image data to a low-dimensional vector space for subsequent calculations; the position encoding layer adds position information to each image block so that the network can perceive the spatial structure of the image; the Transformer Encoder layer is the core part of the model, including multiple self-attention sublayers and feedforward neural network sublayers, which are used to extract image features and integrate global information; the output layer maps the processed feature vector back to the original image size to generate the final virtual enhanced CT image.

7. The method for generating virtual enhanced CT based on a visual self-attention neural network according to claim 6, characterized in that: In the VIT neural network model based on the visual self-attention mechanism, the implementation of the self-attention mechanism includes: Calculate the attention score: In the self-attention sublayer of the Transformer Encoder layer, for each image patch vector, calculate the attention score between it and other image patch vectors; the specific calculation process is to obtain the query, key, and value vectors through linear transformation of the image patch vectors, and then calculate the attention score between the query and the key through the dot product operation. The query of image patch vectors: and The image patch vector : , attention score ,in, Indicates The query vector of the image patch vector The transpose of Attention score normalization: Normalize the calculated attention score and use the softmax function to obtain the normalized attention weight. ,in, is the total number of image blocks, is the normalized attention weight; the degree of attention of each image block vector to other image block vectors is quantified as a weight value between 1 and 1, and the sum of all weights is 1; Weighted summation: According to the normalized attention weight, the value vector of each image block vector is weighted summed to obtain the output of the self-attention layer; that is, ,in, For the The output of the image block vector after the self-attention mechanism, For the The value vector of image patch vectors.

8. The method for generating virtual enhanced CT based on a visual self-attention neural network according to claim 1, characterized in that: In the similarity evaluation and optimization, the statistical method includes calculating the mean absolute error MAE, root mean square error RMSE, mean error ME, structural similarity SSIM, peak signal-to-noise ratio PSNR and Dyss similarity coefficient DSC between the synthesized virtual enhanced CT and the real CT. The MAE calculation formula is: ;in is the number of image pixels, and The virtual enhanced CT and real CT are The gray value of a pixel, For virtual enhanced CT, It is the real CT; The calculation formula is: ; The calculation formula is ; The SSIM calculation formula is: ;in, and They are virtual enhanced CT and real CT images, and is the image mean, and is the image standard deviation, is the covariance, , , is the pixel value range, , ; The PSNR calculation formula is: ;in is the number of bits of the image, and MSE is the mean square error, that is: ; The higher the accuracy of virtual enhanced CT, , , The smaller, , , The bigger.

9. The method for generating virtual enhanced CT based on a visual self-attention neural network according to claim 8, characterized in that: The similarity evaluation and optimization also includes dosimetric evaluation. The dosimetric evaluation is based on the RayStation radiotherapy planning system. The radiotherapy plan is designed for the test case based on the real CT. Then the plan is copied to the virtual enhanced CT for dose calculation. The dose distribution of the real CT and the virtual enhanced CT are obtained respectively. The evaluation indicators include MAE, RMSE, ME and gamma pass rate. The gamma pass rate calculation formula is: ； in, The difference in dosage is the dose tolerance, The distance difference is the distance tolerance, Number of points to meet dose and distance tolerances is the total number of points. When the virtual enhanced CT dose distribution is closer to the real CT dose distribution, the MAE, RMSE, and ME are smaller, and the gamma pass rate is greater.

Images