A processing method and system for predicting lung function based on PET / CT multi-modal images

Abstract

The application discloses a kind of based on PET / CT multimodal image prediction lung function processing method and system, belong to intelligent medical technology field.It includes the following steps: image data is obtained from database and is preprocessed, obtains the image data of input Siamese network;Siamese network is constructed, and the extraction of multimodal feature is carried out to image data;Feature fusion module is set at the end of Siamese network to realize the fusion to the extracted multimodal feature, and obtain multimodal feature vector;The multimodal feature vector after fusion is as the input of prediction model, and prediction result is generated by prediction model.Compared with prior art, the beneficial aspects of the present application are that the limitations of conventional methods relying on breath holding can be effectively overcome, and accurate lung function prediction can still be achieved even with free-breathing CT or breath-hold CT image input alone.

Description

Technical Field

[0001] This invention relates to the field of smart medical technology, and more specifically, to a method and system for predicting lung function based on PET / CT multimodal images. Background Technology

[0002] Lung cancer is a malignant tumor with high incidence and mortality rates worldwide, and preoperative and postoperative pulmonary function assessment is crucial in its treatment. For lung cancer patients, preoperative pulmonary function prediction not only helps doctors develop personalized surgical plans and ensure surgical safety, but also provides valuable references for postoperative recovery planning, complication prevention, and follow-up management. However, traditional pulmonary function testing methods, such as forced vital capacity (FVC) and force expiratory volume in one second (FEV1 / FVC), heavily rely on the patient's subjective cooperation and breathing, which is particularly difficult for elderly, frail, or severely ill lung cancer patients. These patients often cannot complete the tests smoothly due to physical weakness or breathing limitations, resulting in inaccurate or incomplete pulmonary function assessment results, thus affecting surgical decisions and postoperative management. In addition, traditional pulmonary function tests also have the problem of not being convenient for continuous postoperative monitoring. Patients often face challenges such as pain and shortness of breath after surgery, making it difficult to repeat pulmonary function tests. This limits doctors' timely understanding of the patient's pulmonary function recovery and affects the adjustment of subsequent treatment plans.

[0003] PET / CT, as an imaging tool integrating functional metabolic imaging and anatomical imaging, plays a crucial role in the accurate diagnosis, staging, efficacy assessment, and prognostic monitoring of lung cancer. PET, or Positron Emission Computed Tomography, provides images that reflect the metabolic activity of tumor tissue, helping to identify active areas and potential metastases. CT, or Computed Tomography, is a medical imaging technique that provides high-resolution anatomical information, aiding in the precise delineation of tumor boundaries and surrounding organ structures. For lung cancer patients who have undergone preoperative PET / CT, using this imaging data for lung function prediction can reduce the burden of additional examinations and provide a non-invasive and convenient means of lung function assessment. However, current PET / CT methods still face challenges in preoperative lung function prediction. Especially under preoperative conditions, many lung cancer patients cannot complete prolonged breath-holding due to respiratory limitation or physical weakness, which severely affects the quality of PET / CT images and thus limits the accuracy of lung function prediction. Furthermore, existing PET / CT image analysis methods mainly focus on the tumor itself and lack effective algorithms and models for lung function assessment, making the direct extraction of lung function information from PET / CT images a technical challenge.

[0004] In related technologies, such as Chinese patent CNCN123456789A, a CT image-based lung function assessment method is provided. This method indirectly assesses lung function by analyzing the density distribution and volume changes in lung CT images. However, this method does not consider the metabolic information provided by PET images and requires significant breath-holding ability, making it unsuitable for preoperative lung function prediction. While it has promoted the development of lung function assessment technology to some extent, it has not provided any technical inspiration on how to combine PET / CT images for non-invasive and convenient preoperative lung function prediction and overcome limitations such as breath-holding difficulty. Summary of the Invention

[0005] 1. Technical problems to be solved

[0006] To address the problem of accurately predicting preoperative lung function in lung cancer patients in existing technologies, this invention provides a processing method and system for predicting lung function based on PET / CT multimodal imaging. It can leverage the complementary advantages of PET and CT images, and through image processing and machine learning algorithms, accurately predict preoperative lung function in lung cancer patients, while reducing reliance on breath-holding, thus providing a more reliable and convenient means of lung function assessment for personalized treatment of lung cancer.

[0007] 2. Technical Solution

[0008] The objective of this invention is achieved through the following technical solutions.

[0009] The summary section of this application is intended to provide a brief overview of the concepts, which will be described in detail in the detailed description section below. This summary section is not intended to identify key or essential features of the claimed technical solutions, nor is it intended to limit the scope of the claimed technical solutions.

[0010] Some embodiments of this application propose a processing method and system for predicting lung function based on PET / CT multimodal images, in order to solve the technical problems mentioned in the background section above.

[0011] As a first aspect of this application, some embodiments of this application provide a method for predicting lung function based on PET / CT multimodal images, comprising the following steps: obtaining image data from a database and preprocessing it to obtain image data input to a Siamese network; constructing a Siamese network to extract multimodal features from the image data; setting a feature fusion module at the end of the Siamese network to fuse the extracted multimodal features to obtain a multimodal feature vector; using the fused multimodal feature vector as input to a prediction model, and generating a prediction result through the prediction model.

[0012] Furthermore, the imaging data includes CT imaging data and PET imaging data, with the CT imaging data including CT images of free breathing and CT images of breath-holding.

[0013] Furthermore, the preprocessing process includes image format conversion, during which metadata in the image data is preserved. Image format conversion includes NIfTI format conversion, which uses a direction cosine matrix for spatial coordinate transformation. The process is as follows:

[0014] T′=M×T;

[0015] T represents the original spatial coordinates, M is the direction cosine matrix, and T′ is the transformed spatial coordinates.

[0016] Furthermore, the preprocessing also includes discrepancy elimination. Linear normalization is used for CT image data, and standardization is used for PET image data. The process includes: traversing the CT image data to find the minimum pixel value min(f). CT ) and maximum value max(f CT For each pixel value f in the CT image data CT (x), normalize it:

[0017]

[0018] f CT ′(x) represents the original pixel value f CT (x) is the result after mapping to the interval [0, 1] using the linear normalization formula;

[0019] Traverse the PET image data and calculate the mean μ and standard deviation σ of the pixel values;

[0020] For each pixel value f in the PET image data PET (x), normalize:

[0021]

[0022] f PET ′(x) refers to the result after transforming the original pixel value f(x) into a normal distribution with a mean of 0 and a standard deviation of 1 using a standardization formula.

[0023] Furthermore, the Siamese network includes a three-branch structure with shared weights, each branch responsible for processing image data of one modality; the front end of each branch includes four convolutional layers, each using a 3×3 convolutional kernel.

[0024] Furthermore, the fusion of multimodal features includes weighted fusion, a process that involves inputting the feature vector F from the free-breathing CT image data extracted by the Siamese network. CT自由 The feature vector F of breath-hold CT image data CT屏气 and the feature vector F of PET image data PET ;

[0025] By setting learning parameters α, β, and γ, a weighted fusion of multimodal feature vectors is performed to obtain the weighted fusion result value F. 加权融合 :

[0026] F 加权融合 =αF CT自由 +βF CT屏气 +γF PET ;

[0027] Where α+β+γ=1.

[0028] Furthermore, the initial weights of the learning parameters are set to: α = β = γ = 1 / 3. During training, the weights are automatically adjusted based on the feedback from the loss function to optimize the fused multimodal feature vector.

[0029] Furthermore, the fusion of multimodal features also includes concatenating the fused multimodal feature vectors to obtain the final fused feature vector F. 最终融合 :

[0030] F 最终融合=Concat(weighted fusion feature or attention fusion feature, F) CT自由 F CT屏气 F PET );

[0031] F 最终融合 The input is fed into a fully connected layer, where features are compressed and abstracted using the non-linear activation function ReLU to obtain the deep fusion feature F. 输出 , as input to the prediction model:

[0032] F 输出 =ReLU(WFC·F 最终融合 +b);

[0033] WFC is the weight matrix of the fully connected layer, and b is the bias term.

[0034] Furthermore, a weighted mean squared error is used as the loss function to measure the contribution of each modal feature to the prediction result, expressed as:

[0035]

[0036] w i y represents the weight of the i-th mode; i These are true lung function values; These are predicted values.

[0037] As a second aspect of this application, some embodiments of this application provide a system for processing a method to predict lung function based on PET / CT multimodal images, including a data module: acquiring image data from a database and preprocessing it to obtain image data input to a Siamese network; a network construction module: constructing a Siamese network and extracting multimodal features from the image data; setting a feature fusion module at the end of the Siamese network to fuse the extracted multimodal features to obtain a multimodal feature vector; and a prediction module: using the fused multimodal feature vector as input to a prediction model and generating a prediction result through the prediction model.

[0038] 3. Beneficial effects

[0039] Compared to existing technologies, the advantages of this invention are as follows: It employs a Siamese neural network architecture, combining multimodal inputs from free-breathing CT, breath-holding CT, and PET images. The Siamese network effectively integrates various imaging features; feature vectors from different modalities are fused at the Siamese network terminals, enabling multidimensional and refined assessment of lung function; the Siamese network can identify and integrate metabolic features from PET images and anatomical features from free-breathing and breath-holding CT images, enhancing the accuracy of the prediction model; through this multimodal feature fusion, this invention not only effectively overcomes the limitations of traditional methods that rely on breath-holding, but also achieves accurate lung function prediction even with only free-breathing or breath-holding CT images as input. This allows the processing method of this invention to adapt to data acquisition limitations under different clinical conditions. Furthermore, the design combining multimodal and single-modal inputs improves applicability under various clinical conditions, capturing lung function-related information more comprehensively and accurately, and providing more robust prediction results, especially under imperfect clinical imaging conditions. Attached Figure Description

[0040] Figure 1 This is a flowchart illustrating a method for predicting lung function based on PET / CT multimodal images in one embodiment of the present invention.

[0041] Figure 2 This is a schematic diagram of data enhancement in one embodiment of the present invention;

[0042] Figure 3 This is a schematic diagram of the contrast loss in one embodiment of the present invention;

[0043] Figure 4 This is a schematic diagram illustrating the process of converting a free-breathing CT image from DICOM format to NIfTI format in one embodiment of the present invention;

[0044] Figure 5 This is a schematic diagram comparing the original and adjusted breath-hold CT images in one embodiment of the present invention;

[0045] Figure 6 This is a schematic diagram comparing the accuracy and loss function curves of experimental scheme A and scheme B in one embodiment of the present invention;

[0046] Figure 7 This is a schematic diagram comparing the accuracy and loss function curves of the Siamese network and the traditional convolutional network during the training process in one embodiment of the present invention.

[0047] Figure 8 This is a schematic diagram illustrating the impact of weight initialization (mean setting) and adaptive weight adjustment on prediction accuracy in one embodiment of the present invention;

[0048] Figure 9This is a schematic diagram illustrating the change in accuracy after incorporating a contrastive learning method in one embodiment of the present invention;

[0049] Figure 10 This is a schematic diagram illustrating the results of lung function prediction using the technical solution of this invention in one embodiment of the invention. Detailed Implementation

[0050] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0051] Combination Figures 1 to 10 The present invention provides a method for predicting lung function based on PET / CT multimodal images, comprising the following steps:

[0052] S1. Acquire image data and preprocess it:

[0053] Image data is obtained from a medical image database and preprocessed to obtain image data that meets the input requirements of the Siamese network.

[0054] Specifically, imaging data is used to store patient information and is usually stored in DICOM (Digital Imaging and Communications in Medicine) format.

[0055] In one specific embodiment, the image data includes CT (Computed Tomography) image data and PET (Positron Emission Tomography) image data. The CT image data includes both free-breathing CT image data and breath-holding CT image data. The information recorded in the image data includes patient ID, scan parameters, and a direction cosine matrix. Scan parameters include scan time, voltage, and current, while the direction cosine matrix describes the spatial orientation of the image.

[0056] To adapt to different deep learning frameworks and models, image data preprocessing is necessary. Preprocessing includes image format conversion, resolution adjustment, slice selection, image registration, and difference elimination. This preprocessing ensures the data meets the input requirements of the Siamese network, improving the stability and prediction accuracy of the deep learning model. Simultaneously, preprocessing helps reduce noise and redundant information in the data, improving data quality and usability.

[0057] In a specific embodiment, the preprocessing process is as follows:

[0058] S101, Image Format Conversion

[0059] DICOM format image data was converted to a standard format suitable for Siamese networks using medical image processing tools.

[0060] In this embodiment, the medical image processing tool can be dcm2nii, and the standard format is NIfTI or PNG. NIfTI format is suitable for three-dimensional processing, and PNG format is suitable for two-dimensional processing.

[0061] First, during image format conversion, metadata from the DICOM file is preserved. This metadata includes the orientation matrix, pixel spacing, and patient information. Ensuring the complete preservation of metadata is crucial for maintaining spatial consistency and dimensional accuracy of the images.

[0062] Secondly, select the appropriate format based on the analysis objective, i.e., whether the subsequent processing requires 3D modeling or 2D analysis. For example, choose the NIfTI format to suit the applicable 3D processing scenario, preserving spatial information and grayscale distribution; or choose the PNG format to suit the applicable 2D processing or scenarios requiring fast loading, thus meeting different needs.

[0063] Specifically, during NIfTI format conversion, a direction cosine matrix is ​​used for spatial coordinate transformation to ensure that the converted image perfectly matches the original image in space. The NIfTI format conversion process is as follows:

[0064] T′=M×T;

[0065] Where T represents the original spatial coordinates, M is the direction cosine matrix, and T′ is the transformed spatial coordinates.

[0066] In one specific embodiment, DICOM format data from free-breathing CT images is converted to NIfTI format using the dcm2nii tool to verify whether the output file retains spatial consistency. After conversion, the pixel distribution of the image data should be consistent with the original data, and the direction cosine matrix should not be shifted.

[0067] like Figure 4 The diagram illustrates the conversion of free-breathing CT image data from DICOM to NIfTI format. Spatial consistency between the converted and original image data was verified by comparing the Volume Information in the 3D Slicer. The image data converted using the dcm2nii tool demonstrates spatial consistency before and after conversion, indicating that the converted NIfTI format image data maintains spatial consistency with the original DICOM format image data, without any offset or distortion.

[0068] S102, Image resolution adjustment

[0069] Image resolution adjustment of image data is performed using bilinear interpolation, which is based on the values ​​of four adjacent pixels and calculates the value of the target pixel through linear interpolation.

[0070] The bilinear interpolation process requires the following steps: determine the range of pixels to be interpolated based on the target resolution and the original resolution; for each target pixel, calculate its relative distance to the original pixel; use the bilinear interpolation formula to calculate the value of the target pixel based on the values ​​of adjacent pixels and the relative distance; perform interpolation calculations on all target pixels to generate new image data.

[0071] In a specific embodiment, the calculation process of bilinear interpolation is as follows:

[0072] f(x,y)=(1-a)(1-b)f(i,j)+a(1-b)f(i+1,j)+(1-a)bf(i,j+1)+abf(i+1,j+1);

[0073] Where (x,y) are the coordinates of the target pixel, (i,j), (i+1,j), (i,j+1) and (i+1,j+1) are the coordinates of the four neighboring pixels around the target pixel, (a,b) is the relative distance of the target point to these four points (ratio in the horizontal and vertical directions), and f(i,j) is the original pixel value.

[0074] Specifically, after adjusting the resolution, the adjusted image data needs to be verified for consistency to ensure its quality meets the requirements of subsequent processing. Visualization tools should be used to check the adjusted image data to observe whether it remains consistent with the original data. Particular attention should be paid to whether details in key areas such as the lungs are clearly discernible.

[0075] In one specific embodiment, the breath-hold CT image data was adjusted from 512×512 to 256×256 pixels, and it was verified that the detailed features of the lung structure were not lost.

[0076] In one specific embodiment, the breath-hold CT image data was adjusted from 512×512 pixels to 256×256 pixels, and the following verification was performed:

[0077] (1) Detail Feature Examination: Visualization tools were used to examine the adjusted image data, revealing that the detailed features of the lung structure were well preserved, with no obvious blurring or distortion. For example... Figure 5The image shows a comparison between the original and adjusted breath-hold CT images. The comparison of key areas (such as the lungs and bronchi) between the original and adjusted images demonstrates that the adjusted images retain good detail in these key areas, with no significant changes in signal-to-noise ratio and contrast ratio, effectively ensuring image quality.

[0078] (2) Quantitative evaluation results: The signal-to-noise ratio and contrast ratio of the image data before and after adjustment were calculated, such as... Figure 5 As shown, these indicators remained at a high level before and after the adjustment, indicating that the image quality of the adjusted image data was effectively guaranteed.

[0079] Therefore, through interpolation methods and verification steps, image resolution can be effectively adjusted, ensuring that the adjusted image data meets the requirements of subsequent processing in terms of quality and detail.

[0080] S103, Slice Selection

[0081] CT imaging data typically contains multi-slice sections, which contain rich anatomical and pathological information. Key lung information is highly likely to be concentrated in specific slices. When processing lung CT imaging data, accurately selecting representative slices is crucial for subsequent analysis. The process for selecting representative slices is as follows:

[0082] First, a deep learning model is used to segment CT slices, thereby accurately locating the main regions of the lungs, such as the apex, base, and middle lobes.

[0083] In this embodiment, the deep learning model is a convolutional neural network. Segmentation methods include, but are not limited to, threshold segmentation and region growing.

[0084] Specifically, threshold segmentation filters out lung regions based on the grayscale value range of lung tissue in CT image data. Region growing, on the other hand, starts from a seed point and gradually expands to regions that meet certain criteria based on grayscale value similarity, thus capturing lung boundaries more accurately.

[0085] Deep learning segmentation: This method uses a trained CNN model to perform pixel-level classification on CT slices, thereby accurately segmenting the lung region. This approach combines image features and contextual information, resulting in higher segmentation accuracy and robustness.

[0086] Secondly, a selection process is performed. Based on the automatic segmentation, slices covering the main lung regions are further selected according to the location of the lungs.

[0087] In a specific embodiment, the selected slice should meet the following conditions:

[0088] (1) It can cover the complete structure of lung lobes and lung segments, ensuring the comprehensiveness of subsequent analysis;

[0089] (2) Avoid artifacts and noise to ensure the clarity and accuracy of the slices;

[0090] (3) Select slices that can reflect the overall structure and pathological features of the lungs, such as areas containing major blood vessels, bronchi and lung parenchyma.

[0091] Specifically, for 3D modeling, 5-10 consecutive slices can be stacked to enhance the expression and visualization of lung features.

[0092] In one specific embodiment, a segmentation model is used to select lung CT slices from free breathing CT image data, which cover complete lung lobe and segment structures, and are ultimately used for lung function feature extraction.

[0093] In one specific embodiment, a segmentation model is used to select lung CT slices from free breathing CT image data. These slices cover complete lung lobe and segment structures and can ultimately be used for lung function feature extraction.

[0094] Therefore, by optimizing steps such as CT image data preprocessing, automatic segmentation of lung regions, and selection of slices, key information in lung CT slices can be selected and utilized more accurately.

[0095] S104. Alignment and Registration

[0096] In this embodiment, for CT-to-CT registration, specifically registration between free-breathing CT and breath-holding CT, a feature-point-based registration method is employed. Feature points include anatomical key points or corner points. First, anatomical key points or corner points are identified and extracted from both CT image data sets. These feature points are typically located in prominent positions within lung structures, such as the apex, base, and bifurcation of major bronchi. Subsequently, a transformation matrix (including rotation and translation transformations) is calculated to achieve precise alignment of the two CT image data sets. This process ensures a high degree of anatomical consistency between free-breathing CT and breath-holding CT. In the PET-CT registration process, due to the relatively low resolution and large dynamic range of PET images, a mutual information registration method is used for calculation.

[0097] Specifically, mutual information registration assesses the similarity between CT and PET images by calculating their mutual information values. A higher mutual information value indicates a better match between the two images. Therefore, by optimizing the mutual information value, the transformation parameters of the PET images (such as rotation, translation, and scaling) are continuously adjusted until the optimal alignment is achieved. This process improves the spatial consistency between PET and CT image data.

[0098] In one specific embodiment, the above steps enable the alignment of PET and CT using the mutual information registration method. Through meticulous adjustment and optimization of this process, a high degree of consistency between the images of each modality in the lung region can be achieved, which also verifies the effectiveness of the mutual information registration method in PET and CT registration.

[0099] S105, Eliminating Differences

[0100] To eliminate processing inconsistencies caused by differences in pixel value ranges between different modalities, this embodiment performs image data normalization to ensure the consistency and effectiveness of image data from different modalities in subsequent processing.

[0101] Specifically, the pixel values ​​of CT image data typically have a relatively fixed range, but different scanning parameters or devices may cause differences in the pixel value range. To eliminate this difference and facilitate subsequent processing, this embodiment employs a linear normalization method to ensure the consistency and effectiveness of different modal image data in subsequent processing. The process is as follows:

[0102] First, iterate through the entire CT image data to find the minimum pixel value min(f). CT ) and maximum value max(f CT );

[0103] For each pixel value f in the CT image data CT (x), normalized using the following formula:

[0104]

[0105] Among them, f CT ′(x) represents the original pixel value f CT (x) is the result after mapping to the interval [0, 1] using the linear normalization formula.

[0106] Therefore, the normalized pixel value f CT ′(x) will fall within the interval [0, 1].

[0107] More specifically, PET image data typically exhibits a large dynamic range in pixel values, and the distribution of pixel values ​​may differ significantly under different scanning conditions. To address this difference, this embodiment employs a standardization method to normalize the pixel values ​​of PET image data to a standard normal distribution (mean 0, standard deviation 1). The specific process is as follows:

[0108] First, iterate through the entire PET image data and calculate the mean μ and standard deviation σ of the pixel values;

[0109] For each pixel value f in the PET image dataPET (x), normalized using the following formula:

[0110]

[0111] Among them, f PET ′(x) refers to the result after transforming the original pixel value f(x) into a normal distribution with a mean of 0 and a standard deviation of 1 using a standardization formula.

[0112] Therefore, the normalized pixel value f PET ′(x) will conform to a standard normal distribution, i.e., the mean is 0 and the standard deviation is 1.

[0113] In deep learning models (such as convolutional neural networks for feature extraction), normalization is crucial for improving model stability and convergence speed. Especially for multimodal image data (such as CT and PET), normalization can eliminate differences in pixel value ranges between different modalities, enabling deep learning models to process data from different modalities more consistently. Normalizing the pixel values ​​of PET image data to a normal distribution with a mean of 0 and a standard deviation of 1 ensures consistent processing within deep learning networks; similarly, for CT image data, using linear normalization to map pixel values ​​to the [0, 1] interval reduces differences between different modalities. Through normalization, the performance and stability of deep learning models when processing multimodal image data can be effectively improved.

[0114] S106, Data Augmentation

[0115] Data augmentation is employed to randomly transform image data, thereby improving the generalization ability of deep learning models. The data augmentation process includes random rotation, flipping, and random scaling. Specifically, random rotation refers to rotating the image data at random angles to help the deep learning model learn the feature representations of the image data at different angles; flipping includes horizontal and vertical flipping, which increases the diversity of image data; random scaling involves randomly scaling the image data, with a scaling ratio of 90%-110% in this embodiment to simulate images captured at different distances. This helps the deep learning model learn the feature representations of the image data at different scales, thus enhancing the robustness of the deep learning model to different image data acquisition conditions.

[0116] S107, Slice Stacking and Multidimensional Input

[0117] Selected two-dimensional slices are stacked as three-dimensional input to capture the three-dimensional structural features of organs; at the same time, multimodal input data is constructed by combining the 3D volume information of PET image data.

[0118] Specifically, the process of slice stacking and multi-dimensional input is as follows:

[0119] First, the selected two-dimensional slices are stacked according to their spatial order to form a three-dimensional volumetric data. This three-dimensional data preserves the spatial relationships between the slices, thus reflecting the three-dimensional structure of the organ.

[0120] In addition, by combining the 3D volumetric information of PET image data to construct multimodal input data, PET image data provides additional information about organ function (such as metabolic activity), which complements the structural information of CT image data.

[0121] Then, the constructed 3D input data and multimodal input data are used as input to the deep learning model.

[0122] The preprocessed image data meets the requirements of deep learning models in terms of format, resolution, slice selection, registration, and normalization. The preprocessing process ensures the quality and usability of the image data, providing strong support for subsequent deep learning model training and prediction.

[0123] S2 and Siamese network feature extraction:

[0124] A Siamese network is constructed to extract features from preprocessed image data, thereby extracting representative multimodal features.

[0125] Specifically, the Siamese network consists of three branches with shared weights, each corresponding to a different modality of image data (breath-free CT images, breath-hold CT images, and PET images). The front end of each branch in the Siamese network comprises multiple convolutional layers used to extract edge, texture, and morphological features from the images. This three-branch structure with shared weights forms the core of the Siamese network; each branch shares parameters starting from the convolutional layers, ensuring consistency in the feature space.

[0126] In this embodiment, four convolutional layers are set to capture multi-scale spatial features, and the convolutional layers use 3×3 convolutional kernels; in addition, max pooling layers are used to reduce the dimensionality of the features to reduce computational complexity and retain the main spatial information, thereby enhancing the robustness of the features.

[0127] More specifically, to fuse features from different modalities, this embodiment incorporates a Feature Fusion Module (MLP) at the end of the Siamese network. This module extracts and fuses multimodal features, ensuring consistency across different modalities and enabling the Siamese network to learn the correlations between them. The feature vectors fused in the feature fusion layers mentioned in step S3 are the multimodal features extracted and merged by the feature fusion module.

[0128] In one specific embodiment, the Siamese network is built using the PyTorch framework. After building the Siamese network, it is trained and optimized. During the training process, the Adam optimizer is used to update the weights of the Siamese network, and the initial learning rate is set to 0.001.

[0129] Specifically, during the training of the Siamese network, the number of training epochs was set to 50 to ensure sufficient convergence. Simultaneously, a contrastive loss function was used to measure the feature similarity between different modalities, and network performance was optimized by weighting the loss contributions of different modalities. The contrastive loss function encourages feature vectors of similar samples to move closer together, while feature vectors of different samples move further apart.

[0130] Based on the constructed Siamese network, the input is preprocessed image data, namely preprocessed free breathing CT image data, breath-holding CT image data and PET image data; the Siamese network outputs the feature vectors extracted from each branch.

[0131] More specifically, data augmentation strategies are employed to optimize the input image data by randomly rotating (0-15 degrees), flipping, and scaling (±10%) the data, thereby improving the Siamese network's adaptability to different image conditions.

[0132] In one specific embodiment, different network parameter configurations were compared and verified during the experiment. Experimental scheme A used 3 convolutional layers and average pooling layers; experimental scheme B, as an optimization scheme, used 4 convolutional layers and max pooling layers. Experimental results show that the structure using 4 convolutional layers and max pooling layers can more effectively capture the spatial features of lung images, thereby improving the performance of the deep learning model. The accuracy of experimental scheme B reached 92.3%, higher than the 87.5% accuracy of experimental scheme A. Figure 6 As shown, the accuracy and loss function curves of experimental scheme A and scheme B are compared. Scheme B (4 convolutional layers and max pooling layers) outperforms Scheme A (3 convolutional layers and average pooling layers) in both accuracy and convergence speed. This demonstrates that experimental scheme B exhibits faster convergence speed and higher accuracy during training, further validating the effectiveness of the optimization scheme.

[0133] In-depth analysis of the experimental results reveals that increasing the number of convolutional layers and employing max-pooling layers significantly improves the network's feature extraction capabilities. For example... Figure 7The image shows a comparison of the accuracy and loss function curves of the Siamese network and traditional convolutional networks during training. It can be seen that the Siamese network (blue curve) exhibits higher accuracy and a smoother loss decrease during training, validating its advantages in feature extraction and convergence speed. According to... Figure 7 The comparative results show that the Siamese network ultimately achieved an accuracy of 95.0%, higher than the 92.3% of the traditional convolutional network. Furthermore, the loss function of the Siamese network exhibited a more stable and rapid decrease during training. These results demonstrate that increasing the number of convolutional layers and optimizing pooling layers can effectively improve network performance. Additionally, operations such as weighted loss functions and data augmentation also contribute to improving the stability and generalization ability of the Siamese network.

[0134] By constructing and experimentally validating the Siamese network, it can be demonstrated that it can effectively extract shared features and modality-specific information from free-breathing CT image data, breath-holding CT image data, and PET image data. These high-quality feature inputs provide a solid foundation for subsequent fusion and prediction. Furthermore, the optimized design of the Siamese network (such as weighted loss functions, data augmentation, increasing the number of convolutional layers, and employing max-pooling layers) further improves the network's stability and generalization ability.

[0135] S3, Multimodal Feature Fusion:

[0136] In step S2, multimodal features are extracted from free-breathing CT image data, breath-holding CT image data, and PET image data using a Siamese network. In subsequent steps, to integrate features from different modalities and further enhance the predictive power of the deep learning model, this invention introduces a feature fusion layer. This layer is responsible for effectively fusing multimodal features to obtain the final multimodal feature vector. The feature fusion layer enables data from different modalities to complement each other, strengthening the deep learning model's ability to comprehensively utilize different information. Multimodal feature fusion includes weighted fusion, attention mechanisms, feature concatenation and fully connected layer processing, and the introduction of a contrastive learning module. The specific process includes:

[0137] S301, Weighted Fusion

[0138] We perform weighted fusion based on the importance of different modal features.

[0139] First, the feature vectors of free breathing CT images, breath-holding CT images, and PET images extracted from the Siamese network are denoted as Fi, respectively. CT自由 F CT屏气 and F PETThese feature vectors will be used as input and fed into the feature fusion layer for weighted fusion processing to obtain the fused multimodal feature vectors.

[0140] Introduce learning parameters α, β, and γ, satisfying α + β + γ = 1; perform weighted fusion on the three multimodal feature vectors to obtain the weighted fusion result value F. 加权融合 Thus, weights are set:

[0141] F 加权融合 =αF CT自由 +βF CT屏气 +γF PET ;

[0142] Specifically, the initial weights are set to the mean: α = β = γ = 1 / 3, and the optimal weights are automatically learned through backpropagation.

[0143] During training, the weighted mean squared error (WMSE) loss function is used to evaluate the prediction error of the deep learning model. This loss function guides the training of the deep learning model by measuring the difference between the predicted and true values, thereby optimizing the fused feature vector. Specifically, the formula for calculating the weighted mean squared error loss function is as follows:

[0144]

[0145] Among them, w i Let y be the weight of the i-th mode. i These are true lung function values. These are the predicted lung function values.

[0146] In a specific embodiment, such as Figure 8 As shown, the impact of weight initialization and adaptive weight adjustment on prediction accuracy is illustrated, further validating the advantages of automatically learned weights. The comparison results between weight initialization (mean setting) and automatic learning demonstrate that adaptive weights can improve prediction accuracy by approximately 5%. This indicates that allowing the network to automatically learn the weight allocation for different modal features can more effectively integrate multimodal information.

[0147] S302, Attention Mechanism

[0148] An adaptive attention mechanism is introduced into the feature fusion layer to dynamically allocate the importance of modal features. The specific implementation process is as follows:

[0149] First, the attention weights are calculated. The attention weight Ai for each modality feature is calculated using the Softmax function, i.e., Ai = Softmax(WiFi). Here, Wi is the attention weight matrix, and Fi is the feature vector of the i-th modality.

[0150] Secondly, based on the calculated attention weights, the feature vectors of each modality are weighted and fused to obtain the fused feature vector F. 注意力融合 The expression is as follows:

[0151]

[0152] The attention mechanism can dynamically select the optimal modal information based on image quality and task requirements. Especially in the presence of noise or incomplete images, this mechanism can more effectively utilize useful modal information, thereby improving prediction accuracy.

[0153] S303, Feature splicing and fully connected layer

[0154] First, feature concatenation is performed, where the fused multimodal feature vectors are concatenated. Depending on the task requirements, weighted fusion or attention fusion can be used to fuse modal features.

[0155] In one specific embodiment, weighted fusion is used to achieve fused modal features. The feature vector obtained by weighted fusion (weighted fused features) is concatenated with the feature vectors of the original free breathing CT image, breath-holding CT image, and PET image to obtain the final fused feature vector F. 最终融合 .

[0156] In one specific embodiment, attention fusion is used to achieve modal feature fusion. The feature vector obtained by attention fusion (attention fusion feature) is concatenated with the feature vectors of the original free breathing CT image, breath-holding CT image, and PET image to obtain the final fused feature vector F. 最终融合 .

[0157] Specifically, the splicing process is represented as follows:

[0158] F 最终融合 =Concat(weighted fusion feature or attention fusion feature, F) CT自由 F CT屏气 F PET );

[0159] Then, the concatenated feature vector is input into a fully connected layer (FC), where feature compression and abstraction are performed using a non-linear activation function. In this embodiment, the non-linear activation function ReLU is used, and the process is expressed as follows:

[0160] F 输出 =ReLU(WFC·F 最终融合 +b);

[0161] Where WFC is the weight matrix of the fully connected layer, and b is the bias term.

[0162] The final output is the extracted deep fusion feature F. 输出 And it serves as the input to the prediction model.

[0163] S304, Multimodal Contrast Learning

[0164] To enhance the similarity and complementarity of features between modalities, contrastive learning is introduced, and the specific process is as follows:

[0165] Construct positive and negative sample pairs. Positive sample pairs come from different modal features of the same object, while negative sample pairs come from features of different objects.

[0166] A contrastive loss function is employed to maximize the similarity of positive sample pairs (i.e., reduce their Euclidean distance) while minimizing the similarity of negative sample pairs (i.e., increase their Euclidean distance). The contrastive loss function is expressed as follows:

[0167] L = (1-Y)·max(0,mD) + Y·D2;

[0168] Where Y is the sample label (positive / negative sample), D is the Euclidean distance between features, and m is the distance threshold.

[0169] The multimodal feature vectors obtained by fusing the above steps are used as input to the lung function prediction model. Combined with the labels of the training data, the prediction model can more effectively utilize the relationship information between different modalities.

[0170] Experimental results show that combining weighted fusion with contrastive learning improves pre-training accuracy by 8.5%, validating the effectiveness and robustness of this approach in enhancing prediction model accuracy. Strengthening the connections between modalities through contrastive learning further improves the accuracy of multimodal feature fusion in lung function prediction. Figure 9 As shown, the accuracy improved significantly during training after incorporating the contrastive learning method.

[0171] S4, Prediction:

[0172] Based on the multimodal feature vectors fused in step S3, the fused feature vectors are used as input to the prediction model to generate the corresponding lung function assessment results. The specific process is as follows:

[0173] In this invention, it is necessary to predict a continuous numerical value (such as a lung function index) from imaging data; therefore, lung function assessment is considered a regression prediction task. Regression tasks require the prediction model to output continuous numerical predictions, not just classification labels.

[0174] To more accurately measure the contribution of different modal features to the prediction results, the weighted mean squared error (WMSE) is used as the loss function, expressed as:

[0175]

[0176] Among them, w i y represents the weight of the i-th modality, indicating the importance of each modality feature to the prediction result; i These are true lung function values; These are the predicted lung function values.

[0177] By adjusting the weight w i By optimizing the model, the prediction model can balance the information contribution of different modal features, thereby ensuring efficient utilization of fused features.

[0178] Specifically, to further enhance the synergy of multimodal features, this embodiment introduces a contrastive loss function to measure the feature similarity between modalities. The expression for the contrastive loss is:

[0179]

[0180] in, is the similarity label for the sample pair; D is the Euclidean distance between features; margin is the threshold used to control the similarity learning range between features. By introducing contrastive loss, the feature consistency among free-breathing CT images, breath-holding CT images, and PET images can be effectively improved, thereby enhancing the predictive performance of the prediction model.

[0181] Specifically, to prevent overfitting of the prediction model and improve its generalization ability, this embodiment adds an L2 regularization term to the loss function. The specific expression is as follows:

[0182]

[0183] Where λ is the regularization coefficient, used to limit the size of the weights, w j These are the weight parameters in the prediction model. By adding an L2 regularization term, we can effectively prevent the prediction model from having excessively large weights, thereby avoiding overfitting.

[0184] In one specific embodiment, to quantitatively evaluate the accuracy of the prediction results, the following indicators were used for quantitative evaluation:

[0185] (1) Mean Squared Error (MSE): Measures the overall difference between the predicted value and the actual value; the smaller the value, the better the prediction effect. It is expressed as follows:

[0186]

[0187] (2) Root Mean Square Error (RMSE): The actual dimension of the reconstruction error; the closer the value is to 0, the better the prediction effect. It is expressed as follows:

[0188]

[0189] (3) Mean Absolute Error (MAE): Avoids sensitivity to large errors and is suitable for stable error representation. It is expressed as follows:

[0190]

[0191] (4) Coefficient of determination (R) 2 (): Measures the predictive model's ability to explain the variance of the data; the closer the value is to 1, the better the predictive model's performance. It is expressed as follows:

[0192]

[0193] Specifically, a processing system for predicting lung function based on PET / CT multimodal images includes:

[0194] Data module: Retrieves image data from the database and preprocesses it to obtain image data for input into the Siamese network;

[0195] Network construction module: Construct a Siamese network to extract multimodal features from image data; Set a feature fusion module at the end of the Siamese network to fuse the extracted multimodal features and obtain a multimodal feature vector;

[0196] Prediction module: The fused multimodal feature vectors are used as input to the prediction model, which then generates the prediction results.

[0197] In one specific embodiment, the prediction model is trained and validated using free-breathing CT image data, breath-holding CT image data, and PET image data. The prediction results generated through the above steps demonstrate that the prediction model has high prediction accuracy and stability, and can effectively assess the patient's lung function. Through steps such as weighted regression prediction modeling, introducing contrast loss, regularization and overfitting prevention, and prediction result evaluation, accurate prediction and assessment of lung function can be achieved.

[0198] To facilitate intuitive interpretation of the prediction results, this embodiment presents the lung function prediction results in the form of scores, charts, and visualization analysis graphs. Simultaneously, it outputs the contribution weights of each modality feature to explain the importance of different modality features in the prediction results. Figure 10 The results of lung function prediction using this technical solution are shown. The left side shows the input data of the Siamese network (including free breathing CT, breath-holding CT, and PET images). The right side shows a comparison between the predicted lung function indicators (such as FVC and FEV1) and the actual values, further demonstrating the accuracy and robustness of the lung function prediction model.

[0199] The technical solution of this invention employs a Siamese neural network architecture, combining multimodal inputs from free-breathing CT, breath-holding CT, and PET images, and effectively integrates various imaging features through the Siamese network. By fusing feature vectors from different modalities at the Siamese network terminals, it achieves multidimensional and refined assessment of lung function. This effectively overcomes the limitations of traditional methods that rely on breath-holding, and can still achieve accurate lung function prediction even with only free-breathing CT or breath-holding CT images as input.

[0200] The invention and its embodiments have been described above illustratively. This description is not restrictive, and the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. The accompanying drawings are only one embodiment of the invention, and the actual structure is not limited thereto. No reference numerals in the claims should limit the scope of the claims. Therefore, if a person skilled in the art is inspired by this description and designs a similar structure and embodiment without departing from the spirit of the invention, such design should fall within the scope of protection of this patent. Furthermore, the word "comprising" does not exclude other elements or steps, and the word "a" preceding an element does not exclude the inclusion of "a plurality" of that element. Multiple elements stated in the product claims may also be implemented by a single element through software or hardware. The terms "first," "second," etc., are used to indicate names and do not indicate any specific order.

Claims

1. A method for predicting lung function based on PET / CT multimodal images, comprising the following steps: Image data is obtained from the database and preprocessed to obtain image data for input into the Siamese network; A Siamese network is constructed to extract multimodal features from image data. The Siamese network includes a three-branch structure with shared weights, each branch is responsible for processing one modality of image data. The front end of each branch includes four convolutional layers with 3×3 convolutional kernels. A feature fusion module is set at the end of the Siamese network to fuse the extracted multimodal features, resulting in a multimodal feature vector. The fusion of the multimodal features includes weighted fusion, and the process includes: inputting the feature vector of the free breathing CT image data extracted by the Siamese network. CT自由 Feature vectors of breath-hold CT image data CT屏气 and the feature vectors of PET image data PET Set learning parameters , and The multimodal feature vectors are weighted and fused to obtain the weighted fusion result value: ： = CT自由 + CT屏气 + PET ； in, + + = 1; the initial weights of the learning parameters are set as follows: = = = 1 / 3; The fused multimodal feature vectors are concatenated to obtain the final fused feature vector. : = Concat(weighted fusion features, CT自由 , CT屏气 , PET ); Will The input is fed into a fully connected layer, where features are compressed and abstracted using the non-linear activation function ReLU to obtain deep-level fused features. , as input to the prediction model: = ReLU( FC + b); FC is the weight matrix of the fully connected layer, and b is the bias term; Will As input to the prediction model, the prediction results are generated by the prediction model.

2. The method for predicting lung function based on PET / CT multimodal images according to claim 1, characterized in that: The image data includes CT image data and PET image data. The CT image data includes CT image data of free breathing and CT image data of breath-holding.

3. The method for predicting lung function based on PET / CT multimodal images according to claim 2, characterized in that: The preprocessing process includes image format conversion, during which metadata in the image data is preserved; the image format conversion includes NIfTI format conversion, which uses a direction cosine matrix for spatial coordinate transformation, and the process is represented as follows: ； Represents the original spatial coordinates. Here is the direction cosine matrix. ′ represents the transformed spatial coordinates.

4. The method for predicting lung function based on PET / CT multimodal images according to claim 2, characterized in that: The preprocessing process also includes eliminating differences. For CT image data, a linear normalization method is used, and for PET image data, a standardization method is used. The process includes: Traverse the CT image data to find the minimum pixel value. and maximum value ; For each pixel value in CT image data Normalization is performed: ； This indicates that the original pixel values The result after mapping to the [0, 1] interval using the linear normalization formula; Traverse the PET image data and calculate the average pixel value. and standard deviation ; For each pixel value in PET image data Normalization is performed: ； This refers to the original pixel values f ( x The result is obtained by transforming the result into a normal distribution with a mean of 0 and a standard deviation of 1 using a standardization formula.

5. The method for predicting lung function based on PET / CT multimodal images according to claim 1, characterized in that: The weighted mean squared error is used as the loss function to measure the contribution of each modal feature to the prediction result, expressed as: ； For the first The weights of each modality; These are true lung function values; This is a predicted value; The number of modes. =3.

6. A system based on the processing method for predicting lung function based on PET / CT multimodal images according to any one of claims 1 to 5, characterized in that: Includes a data module: which retrieves image data from the database and preprocesses it to obtain image data for input into the Siamese network; Network Construction Module: Constructs a Siamese network to extract multimodal features from image data. The Siamese network includes a three-branch structure with shared weights, each branch handling one modality of image data. Each branch's front end includes four convolutional layers with 3×3 kernels. A feature fusion module is placed at the end of the Siamese network to fuse the extracted multimodal features, obtaining a multimodal feature vector. The multimodal feature fusion includes weighted fusion, the process of which involves inputting the feature vector from the free-breathing CT image data extracted by the Siamese network. CT自由 Feature vectors of breath-hold CT image data CT屏气 and the feature vectors of PET image data PET Set learning parameters , and The multimodal feature vectors are weighted and fused to obtain the weighted fusion result value: ： = CT自由 + CT屏气 + PET ； in, + + = 1; the initial weights of the learning parameters are set as follows: = = = 1 / 3; The fused multimodal feature vectors are concatenated to obtain the final fused feature vector. : = Concat(weighted fusion features, CT自由 , CT屏气 , PET ); Will The input is fed into a fully connected layer, where features are compressed and abstracted using the non-linear activation function ReLU to obtain deep-level fused features. , as input to the prediction model: = ReLU( FC + b); FC is the weight matrix of the fully connected layer, and b is the bias term; Prediction module: As input to the prediction model, the prediction results are generated by the prediction model.

Images