A method and system for identifying brain metastases of lung cancer based on multimodal data

Abstract

The present invention provides a method and system for identifying lung cancer brain metastasis based on multimodal data. The method obtains a user's clinical pathology data, lung CT image data, and brain multi-parameter MRI image data, and performs data preprocessing on each of them; performs anomaly detection on the preprocessed data to obtain a corresponding anomaly score; weights each anomaly score and normalizes it to obtain a comprehensive anomaly score; determines a prediction threshold for the risk of lung cancer brain metastasis based on a maximized Youden index; compares the comprehensive anomaly score with the prediction threshold to determine a prediction result; when the prediction result shows that the user is a patient with lung cancer brain metastasis, separates the preprocessed brain multi-parameter MRI image data into several regions of preset sizes, and calculates reconstruction errors for each region; maps the reconstruction errors of each region to generate an error map to reflect high-risk areas for lung cancer brain metastasis.

Description

Technical Field

[0001] The present invention belongs to the field of medical technology, and in particular relates to a method and system for identifying brain metastases of lung cancer based on multimodal data. Background Art

[0002] Prophylactic cranial irradiation (PCI) can significantly reduce the incidence of brain metastases, but it may cause side effects such as neurocognitive impairment. Currently, PCI is only applicable to certain high-risk patients, and due to the lack of individualization of the method, there is a risk of over- or under-treatment.

[0003] Currently, some studies have attempted to combine clinicopathological features, serum tumor markers, and lung CT imaging data to identify individuals at high risk for brain metastasis from lung cancer. However, these approaches have several significant limitations. First, existing technologies mostly rely on single-modality data, such as clinicopathological data or lung CT images alone, which cannot fully reflect the multidimensional information about a patient's risk of brain metastasis. Although lung CT can reveal the progression of lung cancer, it does not directly predict the occurrence of brain metastasis. Therefore, relying solely on these data is difficult to achieve accurate prediction. Second, multiparametric brain MRI (mpMRI) is an effective tool for assessing changes in the brain microenvironment but has been underutilized. In fact, before circulating tumor cells reach metastatic sites, tumors may modify the metastatic microenvironment by secreting soluble tumor-derived factors or exosomes to adapt to their growth. Magnetic resonance imaging (MRI), due to its radiation-free nature, high soft tissue resolution, and multi-planar, multiparametric imaging, has become an effective tool for assessing the brain microenvironment.

[0004] However, existing technologies fail to deeply mine the valuable information in mpMRI images, resulting in an inability to accurately predict early signs of brain metastasis. Furthermore, existing machine learning models often rely on manual delineation of regions of interest and feature design, a process that is time-consuming and susceptible to human error, making efficiency and accuracy unreliable. More importantly, existing technologies often fail to accurately identify high-risk areas for brain metastasis, making it difficult for radiation oncologists to provide effective treatment decision support. Summary of the Invention

[0005] Based on this, an embodiment of the present invention provides a method and system for identifying lung cancer brain metastasis based on multimodal data. The system aims to accurately predict high-risk individuals for lung cancer brain metastasis and further locate high-risk areas through deep learning methods, combining clinical data, lung CT images and brain multi-parameter MRI image data.

[0006] A first aspect of an embodiment of the present invention provides a method for identifying brain metastases of lung cancer based on multimodal data, the method comprising:

[0007] Obtain the user's clinical pathology data, lung CT image data, and brain multi-parameter MRI image data, and perform data preprocessing on each;

[0008] Perform anomaly detection on the preprocessed data to obtain the corresponding anomaly score. The autoencoder is used to detect anomalies on the preprocessed clinical pathology data, and the Transformer autoencoder is used to detect anomalies on the preprocessed lung CT image data and the preprocessed brain multi-parameter MRI image data.

[0009] After weighting each type of anomaly score, normalize it to get the comprehensive anomaly score;

[0010] Determine the prediction threshold for the risk of brain metastasis from lung cancer based on maximizing the Youden index;

[0011] Comparing the comprehensive anomaly score with the prediction threshold to determine a prediction result;

[0012] When the prediction result shows that the user is a patient with brain metastasis of lung cancer, the pre-processed multi-parameter MRI image data of the brain is obtained, the pre-processed multi-parameter MRI image data of the brain is divided into several regions of preset sizes, and the reconstruction error is calculated for each region;

[0013] The reconstruction errors of each region are mapped to generate an error map to reflect the high-risk areas of lung cancer brain metastasis.

[0014] Furthermore, the step of mapping the reconstruction errors of each region to generate an error map to reflect the high-risk areas of lung cancer brain metastasis includes:

[0015] Determine the enhancement score based on the reconstruction error and comprehensive anomaly score of each region;

[0016] Determining whether the enhancement score is higher than a threshold;

[0017] If it is determined that the enhancement score is higher than the threshold, the region with the enhancement score higher than the threshold is determined as the final high-risk region for lung cancer brain metastasis.

[0018] Furthermore, in the data preprocessing step, the clinical pathology data is subjected to data cleaning, feature encoding and conversion, outlier detection and replacement, feature selection and dimensionality reduction; the lung CT image data is subjected to denoising, cropping and ROI extraction, resampling and enhancement; and the brain multi-parameter MRI image data is subjected to denoising, registration and resampling.

[0019] Furthermore, in the step of performing anomaly detection on the preprocessed data to obtain a corresponding anomaly score, the expression of the anomaly score is:

[0020]

[0021] in, represents the abnormal score of the i-th sample of the brain multi-parameter MRI image data, represents the input image of the i-th sample of brain multi-parameter MRI image data, represents the reconstructed image of the i-th sample of the brain multi-parameter MRI image data, represents the abnormal score of the i-th sample of lung CT image data, represents the input image of the i-th sample of lung CT image data, represents the reconstructed image of the i-th sample of lung CT image data, represents the abnormal score of the i-th sample of clinical pathology data, represents the input features of the i-th sample of clinical pathology data, Represents the reconstructed features of the i-th sample of clinical pathology data.

[0022] Furthermore, in the step of weighting the various anomaly scores and then normalizing them to obtain a comprehensive anomaly score, the weighted expression of the various anomaly scores is:

[0023]

[0024] The expression of the comprehensive anomaly score is:

[0025]

[0026] in, 、 、 are the weight coefficients of the abnormal scores of brain multi-parameter MRI imaging data, lung CT imaging data, and clinical pathology data, respectively. is the abnormal score of brain multi-parameter MRI image data, is the abnormal score of lung CT image data, is the abnormal score of clinical pathological data, is the weighted result, is the minimum value of the weighted result, is the maximum value of the weighted result, is the comprehensive anomaly score.

[0027] Furthermore, in the step of determining the prediction threshold of the risk of brain metastasis of lung cancer based on maximizing the Youden Index, by calculating the sensitivity and specificity under different thresholds, the threshold that maximizes the Youden Index is selected as the final prediction threshold.

[0028] Furthermore, in the step of determining the enhancement score based on the reconstruction error and the comprehensive anomaly score of each region, the expression of the enhancement score is:

[0029]

[0030] in, Indicates the enhancement score, represents the abnormality score of the kth region of the brain multi-parameter MRI image data, represents the weighted result of the kth region, is the weight coefficient of the abnormal score of the kth region of the brain multi-parameter MRI image data, is the weight coefficient of the weighted result of the kth region.

[0031] A second aspect of an embodiment of the present invention provides a system for identifying brain metastases from lung cancer based on multimodal data, for implementing the method for identifying brain metastases from lung cancer based on multimodal data provided in the first aspect. The system includes:

[0032] The preprocessing module is used to obtain the user's clinical pathology data, lung CT image data, and brain multi-parameter MRI image data, and perform data preprocessing respectively;

[0033] Anomaly detection module, which is used to perform anomaly detection on the preprocessed data and obtain the corresponding anomaly score. The autoencoder is used to perform anomaly detection on the preprocessed clinical pathology data, and the Transformer autoencoder is used to perform anomaly detection on the preprocessed lung CT image data and the preprocessed brain multi-parameter MRI image data.

[0034] The normalization processing module is used to weight various anomaly scores and perform normalization processing to obtain a comprehensive anomaly score;

[0035] A first determination module is used to determine a prediction threshold for the risk of brain metastasis of lung cancer based on a maximized Youden index;

[0036] a second determination module, configured to compare the comprehensive anomaly score with the prediction threshold to determine a prediction result;

[0037] a calculation module, configured to obtain pre-processed multi-parameter MRI brain image data when the prediction result shows that the user is a patient with brain metastasis of lung cancer, divide the pre-processed multi-parameter MRI brain image data into a plurality of regions of preset sizes, and calculate the reconstruction error for each region;

[0038] The mapping module is used to map the reconstruction errors of each region and generate an error map to reflect the high-risk areas of lung cancer brain metastasis.

[0039] A third aspect of an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method for identifying brain metastases of lung cancer based on multimodal data provided in the first aspect.

[0040] A fourth aspect of an embodiment of the present invention provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, wherein when the processor executes the program, the method for identifying brain metastases of lung cancer based on multimodal data provided in the first aspect is implemented.

[0041] An embodiment of the present invention provides a method and system for identifying lung cancer brain metastasis based on multimodal data. The method obtains a user's clinical pathology data, lung CT image data, and brain multi-parameter MRI image data, and performs data preprocessing on each of them; performs anomaly detection on the preprocessed data to obtain a corresponding anomaly score; weights each type of anomaly score and normalizes it to obtain a comprehensive anomaly score; determines a prediction threshold for the risk of lung cancer brain metastasis based on the maximized Youden index; compares the comprehensive anomaly score with the prediction threshold to determine a prediction result; when the prediction result shows that the user is a patient with lung cancer brain metastasis, the preprocessed brain multi-parameter MRI image data is divided into several regions of preset sizes, and reconstruction errors are calculated for each region; the reconstruction errors of each region are mapped to generate an error map to reflect high-risk areas for lung cancer brain metastasis. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 This is a flowchart of a method for identifying brain metastases from lung cancer based on multimodal data, according to the first embodiment of the present invention;

[0043] Figure 2 This is a structural block diagram of a lung cancer brain metastasis identification system based on multimodal data provided in Example 2 of the present invention;

[0044] Figure 3 This is a structural block diagram of an electronic device provided in Example 3 of the present invention. DETAILED DESCRIPTION

[0045] To facilitate understanding of the present invention, the present invention will be described more fully below with reference to the accompanying drawings. The drawings illustrate several embodiments of the present invention. However, the present invention may be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and comprehensive understanding of the present invention.

[0046] It should be noted that when an element is referred to as being "fixed to" another element, it may be directly on the other element or there may be an intermediate element. When an element is referred to as being "connected to" another element, it may be directly connected to the other element or there may be an intermediate element. The terms "vertical," "horizontal," "left," "right," and similar expressions used herein are for illustrative purposes only.

[0047] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one skilled in the art to which this invention pertains. The terms used in this specification of the present invention are for the purpose of describing specific embodiments only and are not intended to limit the present invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0048] Example 1

[0049] See also Figure 1 , Figure 1 A flowchart of an implementation of a method for identifying brain metastases of lung cancer based on multimodal data provided in the first embodiment of the present invention is shown. The method for identifying brain metastases of lung cancer based on multimodal data specifically includes steps S01 to S07.

[0050] Step S01: Acquire the user's clinical pathology data, lung CT image data, and brain multi-parameter MRI image data, and perform data preprocessing on each of them.

[0051] Specifically, clinical pathology data include at least tumor size, TNM stage, age, gender, family history, and smoking history. Lung CT imaging data is used to reveal information such as the location and size of lung cancer. Brain multi-parameter MRI imaging data is used to assess changes in the brain microenvironment and predict possible brain metastasis risks.

[0052] It should be noted that the clinical pathology data is processed through data cleaning, feature encoding and conversion, outlier detection and replacement, feature selection and dimensionality reduction. Specifically, data cleaning includes:

[0053] Handling missing values: Clinical pathology data may contain missing values. To ensure data integrity, missing values need to be handled. For numerical features, use median filling to avoid the impact of extreme values that may be caused by mean filling. in, is the value after filling, is the original data, median represents the median; categorical features: use the mode to fill, that is, use the value with the highest frequency in the categorical feature to fill the missing value. in, is the categorical feature after filling, is the original data, mode represents the mode; delete invalid data: for samples with too many missing values or records with obvious data entry errors, delete these samples to ensure data quality.

[0054] Feature encoding and conversion include:

[0055] Numerical feature standardization: Standardization is the processing of numerical features so that their mean is 0 and their standard deviation is 1, thereby avoiding the influence of features of different scales on model training. The formula for standardization is: in, is the original data, is the mean of the feature, is the standard deviation of the feature, The data is normalized. After normalization, each feature has the same scale, which helps improve the training effect of the model.

[0056] One-Hot Encoding of Categorical Features: For categorical features (such as gender, tumor stage, etc.), use One-Hot Encoding to convert each category into an independent binary feature. Suppose a feature has categories (e.g. gender has "male" and "female"), then convert it to binary features:

[0057] male:

[0058] female:

[0059] For categorical features , the one-hot encoding is converted to : In this way, categorical data is converted into numerical data, which is convenient for input into machine learning models.

[0060] Outlier detection and replacement include:

[0061] Outlier detection: Outliers may have a negative impact on model training and therefore need to be detected and processed. In this embodiment of the present invention, a box plot method (BoxPlot) is used to detect outliers:

[0062] First, calculate the quartiles of the data ( and ), and interquartile range (IQR): , then, detect outliers using the following formula: in, and are the first and third quartiles, respectively, and IQR is the interquartile range;

[0063] Outlier processing: For detected outliers, the median substitution method can be used to avoid excessively weakening the actual characteristics of the data: in, For the replaced data, is the original data.

[0064] Feature selection and dimensionality reduction:

[0065] Feature selection: Feature selection is the process of selecting the features that have the greatest impact on the target variable from a large number of features. In this embodiment of the present invention, the Pearson correlation coefficient is used to select the features that are most relevant to the target variable (e.g., brain metastasis risk): ,in, and Characteristics and the target variable The value of and is their mean. Pearson correlation coefficient The value range of is [-1, 1], where close to 1 or -1 indicates that the feature is highly correlated with the target variable;

[0066] Dimensionality reduction: When the feature dimension is too high, principal component analysis (PCA) is used for dimensionality reduction. PCA reduces the number of features by mapping the data into a new feature space while trying to maintain the variance of the data: in, is the original data matrix, is the transformation matrix, is the data matrix after dimensionality reduction.

[0067] Furthermore, the lung CT image data is subjected to denoising, cropping, ROI extraction, resampling, and enhancement processing. Specifically, denoising includes:

[0068] Median Filtering: Used to remove salt and pepper noise while preserving edge information. Median filtering replaces pixel values with the median of its neighborhood, which is suitable for removing isolated noise points in an image. The expression is: ,in, is the position in the original image The pixel value of For the location The pixel values of the filtered image are It's a pixel The surrounding neighborhood, for Any point in the neighborhood of ;

[0069] Gaussian Filtering: Smoothes the image and reduces Gaussian noise by weighted averaging the pixel values in the area surrounding the pixel. Gaussian filtering uses the following Gaussian kernel for convolution: in, is the standard deviation, which controls the smoothness of the filter. are pixel coordinates.

[0070] Cropping and ROI extraction:

[0071] Image cropping: Cropping is used to remove irrelevant background areas from the image and focus on the lung area. The cropped area is determined by manual annotation or automated methods to ensure that the size and area of the input image meet the analysis requirements.

[0072] Lung region extraction: Use threshold segmentation methods (such as the Otsu algorithm) to automatically extract the lung region. The Otsu algorithm automatically selects the optimal threshold by minimizing the intra-class variance: in, is the probability density function of the image gray level, m represents the gray level of the image, and the Otsu algorithm selects the optimal threshold by minimizing the intra-class variance to distinguish lung tissue from non-lung tissue.

[0073] Resampling:

[0074] Image resampling: Since the resolution of different CT scans may be different, it is necessary to unify the images to the same resolution, usually 1 mm³. Resampling uses interpolation methods such as cubic interpolation to adjust: in, is the resampling mapping function, is the original CT image, is the resampled image.

[0075] Data augmentation:

[0076] Rotation: Randomly rotate the image to enhance the adaptability to images at different angles;

[0077] Translation: translate the image horizontally and vertically;

[0078] Scaling: Randomly scale images to improve the model's adaptability to different scales;

[0079] Intensity Adjustment: Simulate images under different scanning conditions by adjusting contrast and brightness.

[0080] Furthermore, the multi-parameter MRI image data of the brain is subjected to denoising, registration and resampling. Specifically, Gaussian filter is used to denoise the multi-parameter MRI image data of the brain. The expression is: ,in, is the standard deviation of the Gaussian filter, which controls the smoothness of the filter;

[0081] Alignment to a standard brain template: Use rigid registration to align brain MRI images with a standard brain template (such as the MNI template). Rigid registration methods include translation and rotation to ensure that brain images from different patients are aligned to a unified standard space.

[0082] Resampling after registration: After registration, the image resolution may need to be adjusted. Use the CubicInterpolation method to resample the image to a standard resolution (e.g. 1 mm³).

[0083] Step S02: perform anomaly detection on the preprocessed data to obtain a corresponding anomaly score.

[0084] Among them, the autoencoder is used to detect anomalies in the preprocessed clinical pathology data, and the Transformer autoencoder is used to detect anomalies in the preprocessed lung CT image data and the preprocessed brain multi-parameter MRI image data.

[0085] In this embodiment, an autoencoder is used to perform unsupervised anomaly detection on clinical pathology data to generate anomaly scores.

[0086] Clinicopathological features include but are not limited to the following:

[0087] Numerical features: tumor size (such as maximum diameter, volume), patient age, etc.

[0088] Categorical features, such as TNM stage, gender, family history, and smoking history, have been converted to numerical form through one-hot encoding.

[0089] Input data format: Assume that the clinical pathological feature vector is , whose dimensions are ,in is the number of samples, is the number of features.

[0090] The autoencoder consists of an encoder and a decoder:

[0091] Encoder: Transforms high-dimensional input features Compression to low-dimensional latent space .

[0092] Decoder: From latent space Reconstructing the original input features .

[0093] Encoder structure:

[0094] Input layer: receiving dimensional feature vector .

[0095] Hidden layer: gradually reduce the feature dimension, use the fully connected layer and activation function (such as ReLU): in, is the encoder weight matrix, is the bias term, is the activation function.

[0096] Decoder structure:

[0097] Input layer: receives latent space representation .

[0098] Hidden layer: gradually restore the feature dimension, using a fully connected layer and activation function:

[0099] in, and are the decoder weights and biases respectively.

[0100] Loss function: The autoencoder learns the normal feature distribution by minimizing the reconstruction error between the input and the reconstructed data. The commonly used loss function is the mean square error (MSE): .

[0101] The model is trained using clinical pathological data from normal patients (without brain metastasis). This allows the model to accurately reconstruct the input features of normal samples while introducing significant reconstruction errors in abnormal samples (patients at high risk of brain metastasis). The reconstruction error (abnormality score) is calculated as follows: , calculate the reconstruction error ,The higher the reconstruction error, the more abnormal the clinical characteristics of the sample.

[0102] For preprocessed lung CT image data and preprocessed multi-parameter brain MRI image data, a Transformer Autoencoder was used for anomaly detection. This model is trained on normal samples (non-metastatic patients) to learn features of normal brain images and identify abnormal regions. The Transformer Autoencoder also consists of an encoder and a decoder. By compressing the input data into a latent space and attempting to recover the original data, the autoencoder can learn key information from the data. The anomaly detection task exploits the autoencoder's reconstruction error of the input data. Specifically, a significant difference between the input data and the reconstructed data indicates an anomaly. By incorporating the Transformer's self-attention mechanism, the Transformer Autoencoder can capture longer-range dependencies during the data encoding process, making it particularly suitable for complex imaging data such as brain MRI images.

[0103] The encoder's task is to map the input multi-parameter brain MRI image into a low-dimensional latent space. The Transformer encoder utilizes a self-attention mechanism to capture long-term dependencies in image data, enabling the model to more accurately extract global features from the image. Its structure is as follows:

[0104] 1.1 Input Layer

[0105] The input data is a brain MRI image, assuming the image size is ,in is the height of the image, is the width, is the depth of the image, is the number of channels of the image (e.g., MRI data with multiple parameters may have multiple channels).

[0106] For the input image, we first perform image block embedding and cut the image into multiple image blocks (patches). The size of each block is usually or , each image patch is then flattened and mapped into a high-dimensional space.

[0107] 1.2. Patch Embedding

[0108] The image is divided into multiple image blocks through slicing operations, each image block is flattened into a vector and mapped to a higher-dimensional space through linear transformation. The above steps are performed through a linear transformation matrix Implemented: ,in, is the vector representation of the k-th image block, Also described as the vector representation of the k-th region.

[0109] 1.3 Positional Encoding

[0110] Since the Transformer model itself does not have positional awareness, positional encoding is needed to preserve the spatial position information of each image block. Positional encoding is usually a vector added to the input vector. ,in, is the position encoding vector, is the final vector after adding the position encoding.

[0111] 1.4 Self-Attention

[0112] The core of the Transformer is the self-attention mechanism, which allows each image patch to interact with all other image patches to capture global image information. For each image patch, three vectors are first generated: query, key, and value. in, , , is the learned weight matrix, are query, key, and value vectors. By calculating the similarity between each query vector and the key vector, a weighted value vector is obtained, thereby generating the relationship information between image patches.

[0113] 1.5 Multi-head Self-Attention

[0114] To capture the various relationships between different parts of the image, a multi-head self-attention mechanism is used. By computing multiple self-attention heads in parallel, each head learns a different feature representation. The outputs of each head are then concatenated to form the final self-attention representation: in, represents the output of the h-th self-attention head, is the output transformation matrix.

[0115] 1.6 Feed-Forward Network

[0116] The output of the Transformer encoder is usually passed through a feedforward neural network, which performs a nonlinear transformation on the output of the self-attention. The network contains two fully connected layers and uses the ReLU activation function: in, is the weight matrix, is the bias term, is the input.

[0117] After multiple layers of self-attention and feedforward neural networks, the encoder outputs the latent space representation is a compressed representation of the image. This latent vector contains the key information of the input image and will be fed into the decoder for reconstruction.

[0118] The task of the decoder is to transform the encoder output into a latent space representation Restore to the original image. The decoder usually contains multiple layers of deconvolution (transposed convolution) layers and upsampling layers to restore the image by gradually increasing the spatial resolution. Its structure is as follows:

[0119] 2.1 Input Layer

[0120] The input layer of the decoder is the latent space representation of the encoder output , which is a low-dimensional vector.

[0121] 2.2 Self-Attention Layer

[0122] The decoder uses a self-attention mechanism to process the latent space representation , which helps the model capture the global dependencies of the image and further optimize the image reconstruction process. Its implementation method is the same as the encoder.

[0123] 2.3 Deconvolutional Layer

[0124] In the decoder, deconvolution (also known as transposed convolution) layers are used to reduce the latent space representation to higher dimensions, gradually recovering the spatial resolution of the image.

[0125] 2.4, Activation Function:

[0126] The last layer uses the Sigmoid activation function to limit the decoded image pixel values to the range of [0,1], which is suitable for restoring grayscale images: ;

[0127] Final output of the decoder is the reconstructed image. The goal of the autoencoder is to optimize the model by minimizing the difference between the input image and the reconstructed image (reconstruction error). The commonly used loss function is the mean squared error (MSE): in, is the sample size, is the input image, To reconstruct the image.

[0128] After training, the model is able to reconstruct normal brain MRI images. Anomaly detection is performed by calculating the image reconstruction error (the difference between the input image and the reconstructed image, also known as the anomaly score). A large reconstruction error indicates that the image pattern differs significantly from normal data and is therefore marked as an anomaly.

[0129] It can be understood that the expression of the anomaly score is:

[0130]

[0131] in, represents the abnormal score of the i-th sample of the brain multi-parameter MRI image data, represents the input image of the i-th sample of brain multi-parameter MRI image data, represents the reconstructed image of the i-th sample of the brain multi-parameter MRI image data, represents the abnormal score of the i-th sample of lung CT image data, represents the input image of the i-th sample of lung CT image data, Represents the reconstructed image of the i-th sample of lung CT image data.

[0132] Step S03: After weighting each type of anomaly score, normalize it to obtain a comprehensive anomaly score.

[0133] Specifically, the weighted expressions of various anomaly scores are:

[0134]

[0135] is the abnormal score of brain multi-parameter MRI image data, is the abnormal score of lung CT image data, is the abnormal score of clinical pathological data, 、 、 These are the weighting coefficients for the abnormality scores of multi-parameter brain MRI imaging data, lung CT imaging data, and clinical pathology data. These weighting coefficients are typically selected based on data characteristics or through cross-validation. Generally, abnormality scores from MRI and CT are likely to have a greater impact, so they can be given higher weights, while clinical pathology features can be given relatively lower weights. The specific weighting should be based on the actual data.

[0136] Normalize the weighted anomaly score to the range [0,1] for subsequent threshold selection. Normalization can use minimum-maximum normalization:

[0137]

[0138] S combined is the weighted result, min(S combined ) is the minimum value of the weighted result, max(S combined ) is the maximum value of the weighted result, S combined，norm is the comprehensive anomaly score, which indicates the final abnormality degree of each sample.

[0139] Step S04: determining a prediction threshold for the risk of brain metastasis of lung cancer based on the maximized Youden index.

[0140] It should be noted that by calculating the sensitivity and specificity at different thresholds, the threshold that maximizes the Youden index is selected as the final prediction threshold. , specifically, Youden's Index is defined as: in:

[0141] Sensitivity: The proportion of positive samples correctly identified by the model (i.e., patients with actual lung cancer brain metastases) as positive.

[0142] Specificity: The proportion of negative samples (i.e., patients without lung cancer brain metastases) that the model correctly identifies as negative.

[0143] Select the optimal threshold by trying different thresholds To maximize the Youden index , select an optimal threshold. The calculation method is as follows:

[0144] For each candidate threshold calculate:

[0145] Sensitivity:

[0146] Specificity:

[0147] Among them, TP, TN, FP, and FN are the true positive, true negative, false positive, and false negative of the model prediction results, respectively.

[0148] Step S05: Compare the comprehensive anomaly score with the prediction threshold to determine a prediction result.

[0149] Understandably, when , the patient was predicted to be at high risk of brain metastasis; , the patient was predicted to be at low risk of brain metastasis.

[0150] Step S06: When the prediction result shows that the user is a patient with brain metastasis of lung cancer, the pre-processed multi-parameter MRI image data of the brain is obtained, the pre-processed multi-parameter MRI image data of the brain is divided into a plurality of regions of preset sizes, and reconstruction errors are calculated for each region.

[0151] Specifically, first, the MRI image is segmented into multiple smaller image patches. These regions can be, for example, or The reconstruction error is calculated for each patch. This patch-by-patch approach can effectively identify local high-risk areas.

[0152] For each region k, the reconstruction error is measured by calculating the difference between the input image and the reconstructed image. Reconstruction Error (Anomaly Score) The calculation formula is:

[0153] in, is the kth region of the original MRI image, is the kth region after reconstruction by the autoencoder, is the reconstruction error of the kth region.

[0154] Step S07 : Mapping the reconstruction errors of each region to generate an error map to reflect the high-risk areas of lung cancer brain metastasis.

[0155] The reconstruction error of each region This error is mapped onto the entire image to generate an error map. Regions with large reconstruction errors indicate that their characteristics differ significantly from the normal distribution learned by the model and are therefore considered potentially high-risk areas. These regions may correspond to brain metastases. The model was not exposed to these features during training, making reconstruction difficult and resulting in large errors.

[0156] In some other embodiments of the present invention, in order to further improve the accuracy of high-risk areas, an enhancement score is determined based on the reconstruction error and comprehensive anomaly score of each area. The expression of the enhancement score is:

[0157]

[0158] in, Indicates the enhancement score, represents the abnormality score of the kth region of the brain multi-parameter MRI image data, represents the weighted result of the kth region, is the weight coefficient of the abnormal score of the kth region of the brain multi-parameter MRI image data, is the weight coefficient of the weighted result of the kth region;

[0159] Determining whether the enhancement score is higher than a threshold;

[0160] If it is determined that the enhancement score is higher than the threshold, the region with the enhancement score higher than the threshold is determined as the final high-risk region for lung cancer brain metastasis.

[0161] As can be understood, combining the comprehensive anomaly score with the reconstruction error makes regions with both high comprehensive anomaly scores and high reconstruction error appear to be higher-risk areas. Ultimately, the calculated enhancement score is used to generate a heatmap of high-risk areas. The heatmap of high-risk areas can be displayed using color mapping (e.g., red indicates high-risk areas) to facilitate intuitive identification by doctors.

[0162] The heatmap is superimposed on the original MRI image to highlight high-risk areas. Alpha blending is used to ensure that the heatmap is integrated with the original image, enhancing the focus on high-risk areas.

[0163] For the identified high-risk areas, bounding boxes or other marking methods can be drawn on these areas to further emphasize the locations of these areas.

[0164] In summary, the lung cancer brain metastasis identification method and system based on multimodal data in the above-mentioned embodiments of the present invention obtains the user's clinical pathology data, lung CT image data, and brain multi-parameter MRI image data, and performs data preprocessing respectively; performs anomaly detection on the preprocessed data to obtain the corresponding anomaly score; after weighting each type of anomaly score, normalizes it to obtain a comprehensive anomaly score; determines the prediction threshold of the lung cancer brain metastasis risk based on the maximized Youden index; compares the comprehensive anomaly score with the prediction threshold to determine the prediction result; when the prediction result shows that the user is a lung cancer brain metastasis patient, the preprocessed brain multi-parameter MRI image data is divided into several preset-size regions, and the reconstruction error is calculated separately; the reconstruction error of each region is mapped to generate an error map to reflect the high-risk areas of lung cancer brain metastasis.

[0165] Example 2

[0166] See also Figure 2 , Figure 2 This is a block diagram of a system for identifying brain metastases from lung cancer based on multimodal data according to a second embodiment of the present invention. The system 200 for identifying brain metastases from lung cancer based on multimodal data includes a preprocessing module 21, an anomaly detection module 22, a normalization module 23, a first determination module 24, a second determination module 25, a calculation module 26, and a mapping module 27.

[0167] The preprocessing module 21 is used to obtain the user's clinical pathology data, lung CT image data, and brain multi-parameter MRI image data, and perform data preprocessing on each of them, including data cleaning, feature encoding and conversion, outlier detection and replacement, feature selection, and dimensionality reduction on the clinical pathology data; denoising, cropping, ROI extraction, resampling, and enhancement on the lung CT image data; and denoising, registration, and resampling on the brain multi-parameter MRI image data.

[0168] The anomaly detection module 22 is used to perform anomaly detection on the preprocessed data and obtain a corresponding anomaly score. The anomaly detection is performed on the preprocessed clinical pathology data using an autoencoder, and the anomaly detection is performed on the preprocessed lung CT image data and the preprocessed brain multi-parameter MRI image data using a Transformer autoencoder. Specifically, the expression of the anomaly score is:

[0169]

[0170] in, represents the abnormal score of the i-th sample of the brain multi-parameter MRI image data, represents the input image of the i-th sample of brain multi-parameter MRI image data, represents the reconstructed image of the i-th sample of the brain multi-parameter MRI image data, represents the abnormal score of the i-th sample of lung CT image data, represents the input image of the i-th sample of lung CT image data, represents the reconstructed image of the i-th sample of lung CT image data, represents the abnormal score of the i-th sample of clinical pathology data, represents the input features of the i-th sample of clinical pathology data, Represents the reconstructed features of the i-th sample of clinical pathology data;

[0171] The normalization processing module 23 is used to weight the various anomaly scores and perform normalization processing to obtain a comprehensive anomaly score. The expression for the weighted anomaly scores is:

[0172]

[0173] The expression of the comprehensive anomaly score is:

[0174]

[0175] in, 、 、 are the weight coefficients of the abnormal scores of brain multi-parameter MRI imaging data, lung CT imaging data, and clinical pathology data, respectively. is the abnormal score of brain multi-parameter MRI image data, is the abnormal score of lung CT image data, is the abnormal score of clinical pathological data, is the weighted result, is the minimum value of the weighted result, is the maximum value of the weighted result, is the comprehensive anomaly score;

[0176] A first determination module 24 is configured to determine a prediction threshold for the risk of brain metastasis of lung cancer based on the maximized Youden index, wherein the threshold that maximizes the Youden index is selected as the final prediction threshold by calculating the sensitivity and specificity at different thresholds;

[0177] A second determination module 25 is configured to compare the comprehensive anomaly score with the prediction threshold to determine a prediction result;

[0178] a calculation module 26 configured to obtain pre-processed multi-parameter MRI brain image data when the prediction result indicates that the user is a patient with brain metastasis of lung cancer, divide the pre-processed multi-parameter MRI brain image data into a plurality of regions of preset sizes, and calculate reconstruction errors for each region;

[0179] The mapping module 27 is used to map the reconstruction errors of each region to generate an error map to reflect the high-risk areas of lung cancer brain metastasis.

[0180] Furthermore, in some other embodiments of the present invention, the multimodal data-based lung cancer brain metastasis identification system 200 further includes:

[0181] The third determination module is used to determine the enhancement score based on the reconstruction error and comprehensive anomaly score of each region. The expression of the enhancement score is:

[0182]

[0183] in, Indicates the enhancement score, represents the abnormality score of the kth region of the brain multi-parameter MRI image data, represents the weighted result of the kth region, is the weight coefficient of the abnormal score of the kth region of the brain multi-parameter MRI image data, is the weight coefficient of the weighted result of the kth region;

[0184] A judgment module, configured to judge whether the enhancement score is higher than a threshold;

[0185] The fourth determining module is configured to determine, if it is determined that the enhancement score is higher than a threshold, the region with the enhancement score higher than the threshold as a final high-risk region for lung cancer brain metastasis.

[0186] Example 3

[0187] Another aspect of the present invention provides an electronic device, see Figure 3 , shown is an electronic device in a third embodiment of the present invention, including a memory 20, a processor 10, and a computer program 30 stored in the memory and executable on the processor. When the processor 10 executes the computer program 30, the method for identifying brain metastases of lung cancer based on multimodal data as described above is implemented.

[0188] In some embodiments, the processor 10 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip, used to run program codes or process data stored in the memory 20, such as executing access restriction programs.

[0189] The memory 20 includes at least one type of readable storage medium, including flash memory, a hard disk, a multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the memory 20 may be an internal storage unit of the electronic device, such as the hard disk of the electronic device. In other embodiments, the memory 20 may also be an external storage device of the electronic device, such as a plug-in hard disk equipped on the electronic device, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc. Furthermore, the memory 20 may include both an internal storage unit of the electronic device and an external storage device. The memory 20 can be used not only to store application software and various types of data of the electronic device, but also to temporarily store data that has been output or is about to be output.

[0190] It should be pointed out that Figure 3 The structure shown does not constitute a limitation to the electronic device. In other embodiments, the electronic device may include fewer or more components than shown in the figure, or combine certain components, or arrange the components differently.

[0191] An embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon. When the program is executed by a processor, the method for identifying brain metastases of lung cancer based on multimodal data as described above is implemented.

[0192] Those skilled in the art will appreciate that the logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device), or in conjunction with such instruction execution system, apparatus, or device. For purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by an instruction execution system, apparatus, or device, or in conjunction with such instruction execution system, apparatus, or device.

[0193] More specific examples (a non-exhaustive list) of computer-readable media include the following: an electrical connection with one or more wires (electronic devices), a portable computer disk cartridge (magnetic devices), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disc read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting, or processing it in another suitable manner as necessary, and then storing it in a computer memory.

[0194] It should be understood that various components of the present invention may be implemented using hardware, software, firmware, or a combination thereof. In the aforementioned embodiments, multiple steps or methods may be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one or a combination of the following technologies known in the art may be used: a discrete logic circuit having logic gate circuits for implementing logic functions on data signals, an application-specific integrated circuit having suitable combinational logic gate circuits, a programmable gate array (PGA), a field-programmable gate array (FPGA), etc.

[0195] Throughout this specification, reference to terms such as "one embodiment," "some embodiments," "examples," "specific examples," or "some examples" means that a specific feature, structure, material, or characteristic described in conjunction with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0196] The above embodiments merely illustrate several implementations of the present invention, and while their descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art would be able to make numerous modifications and improvements without departing from the spirit of the present invention, all of which fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be determined by the appended claims.

Claims

1. A method for identifying brain metastases from lung cancer based on multimodal data, characterized in that: The method comprises: Obtain the user's clinical pathology data, lung CT image data, and brain multi-parameter MRI image data, and perform data preprocessing on each; Perform anomaly detection on the preprocessed data to obtain the corresponding anomaly score. The autoencoder is used to detect anomalies on the preprocessed clinical pathology data, and the Transformer autoencoder is used to detect anomalies on the preprocessed lung CT image data and the preprocessed brain multi-parameter MRI image data. After weighting each type of anomaly score, normalize it to get the comprehensive anomaly score; Determine the prediction threshold for the risk of brain metastasis from lung cancer based on maximizing the Youden index; Comparing the comprehensive anomaly score with the prediction threshold to determine a prediction result; When the prediction result shows that the user is a patient with brain metastasis of lung cancer, the pre-processed multi-parameter MRI image data of the brain is obtained, the pre-processed multi-parameter MRI image data of the brain is divided into several regions of preset sizes, and the reconstruction error is calculated for each region; Map the reconstruction errors of each region to generate an error map to reflect the high-risk areas of lung cancer brain metastasis; Determine the enhancement score based on the reconstruction error and comprehensive anomaly score of each region; Determining whether the enhancement score is higher than a threshold; If it is determined that the enhancement score is higher than the threshold, the region with the enhancement score higher than the threshold is determined as the final high-risk region for lung cancer brain metastasis.

2. The method for identifying brain metastases of lung cancer based on multimodal data according to claim 1, characterized in that: In the data preprocessing step, clinical pathology data is subjected to data cleaning, feature encoding and conversion, outlier detection and replacement, feature selection and dimensionality reduction; lung CT image data is subjected to denoising, cropping and ROI extraction, resampling and enhancement; and brain multi-parameter MRI image data is subjected to denoising, registration and resampling.

3. The method for identifying brain metastases of lung cancer based on multimodal data according to claim 2, characterized in that: In the step of performing anomaly detection on the preprocessed data to obtain the corresponding anomaly score, the expression of the anomaly score is: in, represents the abnormal score of the i-th sample of the brain multi-parameter MRI image data, represents the input image of the i-th sample of brain multi-parameter MRI image data, represents the reconstructed image of the i-th sample of the brain multi-parameter MRI image data, represents the abnormal score of the i-th sample of lung CT image data, represents the input image of the i-th sample of lung CT image data, represents the reconstructed image of the i-th sample of lung CT image data, represents the abnormal score of the i-th sample of clinical pathology data, represents the input features of the i-th sample of clinical pathology data, Represents the reconstructed features of the i-th sample of clinical pathology data.

4. The method for identifying brain metastases of lung cancer based on multimodal data according to claim 3, characterized in that: In the step of weighting various types of anomaly scores and then normalizing them to obtain a comprehensive anomaly score, the expression for weighting various types of anomaly scores is: The expression of the comprehensive anomaly score is: in, 、 、 are the weight coefficients of the abnormal scores of brain multi-parameter MRI imaging data, lung CT imaging data, and clinical pathology data, respectively. is the abnormal score of brain multi-parameter MRI image data, is the abnormal score of lung CT image data, is the abnormal score of clinical pathological data, is the weighted result, is the minimum value of the weighted result, is the maximum value of the weighted result, is the comprehensive anomaly score.

5. The method for identifying brain metastases of lung cancer based on multimodal data according to claim 4, characterized in that: In the step of determining the prediction threshold of the risk of brain metastasis of lung cancer based on maximizing the Youden Index, the sensitivity and specificity under different thresholds are calculated, and the threshold that maximizes the Youden Index is selected as the final prediction threshold.

6. The method for identifying brain metastases of lung cancer based on multimodal data according to claim 5, characterized in that: In the step of determining the enhancement score based on the reconstruction error and the comprehensive anomaly score of each region, the expression of the enhancement score is: in, Indicates the enhancement score, represents the abnormality score of the kth region of the brain multi-parameter MRI image data, represents the weighted result of the kth region, is the weight coefficient of the abnormal score of the kth region of the brain multi-parameter MRI image data, is the weight coefficient of the weighted result of the kth region.

7. A lung cancer brain metastasis identification system based on multimodal data, characterized in that: A system for implementing the method for identifying brain metastases of lung cancer based on multimodal data according to any one of claims 1 to 6, comprising: The preprocessing module is used to obtain the user's clinical pathology data, lung CT image data, and brain multi-parameter MRI image data, and perform data preprocessing respectively; Anomaly detection module, which is used to perform anomaly detection on the preprocessed data and obtain the corresponding anomaly score. The autoencoder is used to perform anomaly detection on the preprocessed clinical pathology data, and the Transformer autoencoder is used to perform anomaly detection on the preprocessed lung CT image data and the preprocessed brain multi-parameter MRI image data. The normalization processing module is used to weight various anomaly scores and perform normalization processing to obtain a comprehensive anomaly score; A first determination module is used to determine a prediction threshold for the risk of brain metastasis of lung cancer based on a maximized Youden index; a second determination module, configured to compare the comprehensive anomaly score with the prediction threshold to determine a prediction result; a calculation module, configured to obtain pre-processed multi-parameter MRI brain image data when the prediction result shows that the user is a patient with brain metastasis of lung cancer, divide the pre-processed multi-parameter MRI brain image data into a plurality of regions of preset sizes, and calculate the reconstruction error for each region; A mapping module is used to map the reconstruction errors of each region and generate an error map to reflect the high-risk areas of lung cancer brain metastasis; The third determination module is used to determine the enhancement score based on the reconstruction error and the comprehensive anomaly score of each region; A judgment module, configured to judge whether the enhancement score is higher than a threshold; The fourth determining module is configured to determine, if it is determined that the enhancement score is higher than a threshold, the region with the enhancement score higher than the threshold as a final high-risk region for lung cancer brain metastasis.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method for identifying brain metastases of lung cancer based on multimodal data according to any one of claims 1 to 6 is implemented.

9. An electronic device, characterized in that: The invention comprises a memory, a processor, and a computer program stored in the memory and running on the processor, wherein when the processor executes the program, the method for identifying brain metastasis of lung cancer based on multimodal data according to any one of claims 1 to 6 is implemented.

Images