Lung cancer brain metastasis identification method and system based on multi-modal data

Abstract

The invention provides a lung cancer brain metastasis identification method and system based on multi-modal data, and the method comprises the steps: obtaining clinical pathology data, lung CT image data and brain multi-parameter MRI image data of a user, and carrying out the data preprocessing; performing anomaly detection on the preprocessed data to obtain a corresponding anomaly score; weighting various abnormal scores, and performing normalization processing to obtain a comprehensive abnormal score; determining a prediction threshold value of the brain metastasis risk of the lung cancer according to the maximum Youden index; comparing the comprehensive abnormal score with a prediction threshold, and determining a prediction result; when the prediction result shows that the user is a lung cancer brain metastasis patient, dividing the preprocessed brain multi-parameter MRI image data into a plurality of areas with preset sizes, and respectively calculating reconstruction errors; and mapping the reconstruction error of each region to generate an error mapping graph so as to reflect the high-risk region of the lung cancer brain metastasis.

Description

Technical Field

[0001] The present invention belongs to the field of medical technology, and in particular relates to a method and system for identifying brain metastases of lung cancer based on multimodal data. Background Art

[0002] Prophylactic cranial irradiation (PCI) can significantly reduce the incidence of brain metastases, but it may cause side effects such as neurocognitive impairment. Currently, PCI is only applicable to some high-risk patients, and due to the lack of individualization of the method, there is a risk of over-treatment or under-treatment.

[0003] At present, some studies have attempted to combine clinical pathological characteristics, serum tumor markers and lung CT imaging data to identify individuals at high risk of brain metastasis from lung cancer, but these methods have some obvious shortcomings. First, most existing technologies rely on single-modality data, such as clinical pathological data or lung CT images alone, which cannot fully reflect the multidimensional information of the patient's risk of brain metastasis. Although lung CT can reveal the progression of lung cancer, it has no direct predictive effect on the occurrence of brain metastasis, so it is difficult to achieve accurate prediction by relying solely on these data. Secondly, brain multi-parameter MRI (mpMRI) has not been fully utilized as an effective tool for evaluating changes in the brain microenvironment. In fact, before circulating tumor cells reach the metastatic lesions, tumors may change the microenvironment of the metastatic lesions by secreting soluble tumor-derived factors or exosomes to adapt to their growth. Magnetic resonance imaging (MRI) has become an effective tool for evaluating the brain microenvironment due to its advantages of no radiation, high soft tissue resolution and multi-plane multi-parameter imaging.

[0004] However, existing technologies fail to deeply mine the effective information in mpMRI images, resulting in the inability to accurately predict the early signals of brain metastasis. In addition, most existing machine learning models rely on manual delineation of regions of interest and design of features. The entire process is time-consuming and easily affected by human factors, and efficiency and accuracy cannot be guaranteed. More importantly, existing technologies are usually unable to accurately identify high-risk areas for brain metastasis, making it difficult to provide effective treatment decision support for radiotherapists. Summary of the invention

[0005] Based on this, an embodiment of the present invention provides a method and system for identifying brain metastases from lung cancer based on multimodal data, which aims to accurately predict high-risk individuals for brain metastases from lung cancer and further locate high-risk areas through a deep learning method combined with clinical data, lung CT images, and brain multi-parameter MRI image data.

[0006] A first aspect of an embodiment of the present invention provides a method for identifying brain metastases of lung cancer based on multimodal data, the method comprising: Obtain the user's clinical pathology data, lung CT image data, and brain multi-parameter MRI image data, and perform data preprocessing respectively; Perform anomaly detection on the preprocessed data to obtain the corresponding anomaly score, wherein the preprocessed clinical pathology data is detected by the autoencoder, and the preprocessed lung CT image data and the preprocessed brain multi-parameter MRI image data are detected by the Transformer autoencoder; After weighting various anomaly scores, normalization is performed to obtain a comprehensive anomaly score; Determine the prediction threshold of lung cancer brain metastasis risk based on maximizing the Youden index; Comparing the comprehensive anomaly score with the prediction threshold to determine a prediction result; When the prediction result shows that the user is a patient with brain metastasis of lung cancer, the preprocessed brain multi-parameter MRI image data is obtained, the preprocessed brain multi-parameter MRI image data is divided into a number of regions of preset sizes, and reconstruction errors are calculated respectively; The reconstruction errors of each region are mapped to generate an error map to reflect the high-risk areas of lung cancer brain metastasis.

[0007] Furthermore, the step of mapping the reconstruction errors of each region to generate an error map to reflect the high-risk region of lung cancer brain metastasis includes: Determine the enhancement score based on the reconstruction error and comprehensive anomaly score of each region; Determining whether the enhancement score is higher than a threshold; If it is determined that the enhancement score is higher than the threshold, the region where the enhancement score is higher than the threshold is determined as the final high-risk region for lung cancer brain metastasis.

[0008] Furthermore, in the step of data preprocessing, the clinical pathological data is subjected to data cleaning, feature encoding and conversion, outlier detection and replacement, feature selection and dimensionality reduction; the lung CT image data is subjected to denoising, cropping and ROI extraction, resampling and enhancement; and the brain multi-parameter MRI image data is subjected to denoising, registration and resampling.

[0009] Furthermore, in the step of performing anomaly detection on the preprocessed data to obtain a corresponding anomaly score, the expression of the anomaly score is:

[0010] in, represents the abnormal score of the i-th sample of the brain multi-parameter MRI image data, represents the input image of the i-th sample of brain multi-parameter MRI image data, represents the reconstructed image of the i-th sample of the brain multi-parameter MRI image data, represents the abnormal score of the i-th sample of lung CT image data, represents the input image of the i-th sample of lung CT image data, represents the reconstructed image of the i-th sample of lung CT image data, represents the abnormal score of the i-th sample of clinical pathology data, represents the input features of the i-th sample of clinical pathology data, Represents the reconstructed features of the i-th sample of clinical pathological data.

[0011] Furthermore, in the step of weighting various types of anomaly scores and then normalizing them to obtain a comprehensive anomaly score, the expression for weighting various types of anomaly scores is:

[0012] The expression of the comprehensive anomaly score is:

[0013] in, , , are the weight coefficients of the abnormal scores of brain multi-parameter MRI image data, lung CT image data, and clinical pathology data, respectively. is the abnormal score of brain multi-parameter MRI image data, is the abnormal score of lung CT image data, is the abnormal score of clinical pathological data, is the weighted result, is the minimum value of the weighted result, is the maximum value of the weighted result, is the comprehensive anomaly score.

[0014] Furthermore, in the step of determining the prediction threshold of the risk of brain metastasis of lung cancer based on maximizing the Youden Index, by calculating the sensitivity and specificity under different thresholds, the threshold that maximizes the Youden Index is selected as the final prediction threshold.

[0015] Furthermore, in the step of determining the enhancement score according to the reconstruction error and the comprehensive anomaly score of each region, the expression of the enhancement score is:

[0016] in, Indicates the enhancement score, represents the abnormal score of the kth region of the brain multi-parameter MRI image data, represents the weighted result of the kth region, is the weight coefficient of the abnormal score of the kth region of the brain multi-parameter MRI image data, is the weight coefficient of the weighted result of the kth region.

[0017] A second aspect of an embodiment of the present invention provides a system for identifying brain metastases of lung cancer based on multimodal data, which is used to implement the method for identifying brain metastases of lung cancer based on multimodal data provided in the first aspect. The system includes: The preprocessing module is used to obtain the user's clinical pathology data, lung CT image data, and brain multi-parameter MRI image data, and perform data preprocessing respectively; An anomaly detection module is used to perform anomaly detection on the preprocessed data and obtain the corresponding anomaly score. The preprocessed clinical pathology data is detected by an autoencoder, and the preprocessed lung CT image data and the preprocessed brain multi-parameter MRI image data are detected by a Transformer autoencoder. The normalization processing module is used to weight various anomaly scores and perform normalization processing to obtain a comprehensive anomaly score; A first determination module is used to determine a prediction threshold of the risk of brain metastasis of lung cancer based on maximizing the Youden index; A second determination module, configured to compare the comprehensive anomaly score with the prediction threshold to determine a prediction result; A calculation module, for obtaining pre-processed brain multi-parameter MRI image data when the prediction result shows that the user is a patient with brain metastasis of lung cancer, dividing the pre-processed brain multi-parameter MRI image data into a plurality of regions of preset sizes, and calculating reconstruction errors respectively; The mapping module is used to map the reconstruction errors of each region and generate an error map to reflect the high-risk areas of lung cancer brain metastasis.

[0018] A third aspect of an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method for identifying brain metastases of lung cancer based on multimodal data provided in the first aspect.

[0019] A fourth aspect of an embodiment of the present invention provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, wherein when the processor executes the program, the method for identifying brain metastases of lung cancer based on multimodal data provided in the first aspect is implemented.

[0020] A method and system for identifying brain metastasis of lung cancer based on multimodal data is provided in an embodiment of the present invention. The method obtains clinical pathological data, lung CT image data and brain multi-parameter MRI image data of a user, and performs data preprocessing respectively; performs abnormality detection on the preprocessed data to obtain corresponding abnormality scores; after weighting various abnormality scores, performs normalization processing to obtain a comprehensive abnormality score; determines a prediction threshold of the risk of brain metastasis of lung cancer based on the maximized Youden index; compares the comprehensive abnormality score with the prediction threshold to determine a prediction result; when the prediction result shows that the user is a patient with brain metastasis of lung cancer, the preprocessed brain multi-parameter MRI image data is divided into several regions of preset sizes, and reconstruction errors are calculated respectively; the reconstruction errors of each region are mapped to generate an error mapping diagram to reflect high-risk areas of brain metastasis of lung cancer. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 A flowchart of a method for identifying brain metastases of lung cancer based on multimodal data provided in Example 1 of the present invention; Figure 2 A structural block diagram of a lung cancer brain metastasis identification system based on multimodal data provided in Embodiment 2 of the present invention; Figure 3 This is a structural block diagram of an electronic device provided in Embodiment 3 of the present invention. DETAILED DESCRIPTION

[0022] In order to facilitate the understanding of the present invention, the present invention will be described more fully below with reference to the relevant drawings. Several embodiments of the present invention are given in the drawings. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the disclosure of the present invention more thorough and comprehensive.

[0023] It should be noted that when an element is referred to as being "fixed to" another element, it may be directly on the other element or there may be a central element. When an element is considered to be "connected to" another element, it may be directly connected to the other element or there may be a central element at the same time. The terms "vertical", "horizontal", "left", "right" and similar expressions used herein are for illustrative purposes only.

[0024] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which the present invention belongs. The terms used herein in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "and / or" used herein includes any and all combinations of one or more of the related listed items.

[0025] Embodiment 1 See also Figure 1 , Figure 1 A flowchart of an implementation of a method for identifying brain metastases of lung cancer based on multimodal data provided in Embodiment 1 of the present invention is shown. The method for identifying brain metastases of lung cancer based on multimodal data specifically includes steps S01 to S07.

[0026] Step S01, obtaining the user's clinical pathological data, lung CT image data and brain multi-parameter MRI image data, and performing data preprocessing respectively.

[0027] Specifically, clinical pathological data include at least tumor size, TNM stage, age, gender, family history, and smoking history. Lung CT imaging data is used to reveal information such as the location and size of lung cancer. Brain multi-parameter MRI imaging data is used to evaluate changes in the brain microenvironment and predict possible brain metastasis risks.

[0028] It should be noted that the clinical pathology data is processed by data cleaning, feature encoding and conversion, outlier detection and replacement, feature selection and dimensionality reduction. Specifically, data cleaning includes: Handling missing values: There may be missing values ​​in clinical pathology data. In order to ensure the integrity of the data, missing values ​​need to be handled; Numerical features: Use median filling to avoid the impact of extreme values ​​that may be caused by mean filling. in, is the value after filling, is the original data, median represents the median; categorical features: use the mode to fill, that is, use the value with the highest frequency in the categorical feature to fill the missing value. in, is the categorical feature after filling, is the original data, mode represents the mode; delete invalid data: for samples with too many missing values ​​or records with obvious data entry errors, delete these samples to ensure data quality.

[0029] Feature encoding and conversion include: Numerical feature standardization: Standardization is the processing of numerical features to make their mean 0 and standard deviation 1, thereby avoiding the influence of features of different scales on model training. The formula for standardization is: in, is the original data, is the mean of the feature, is the standard deviation of the feature, is the standardized data. After standardization, each feature has the same scale, which helps improve the training effect of the model; One-hot encoding of categorical features: For categorical features (such as gender, tumor stage, etc.), one-hot encoding is used to convert each category into an independent binary feature. Suppose a feature has categories (e.g., gender has "male" and "female"), then convert it to Binary features: male:

[0030] female:

[0031] For categorical features , the one-hot encoding is converted to : In this way, categorical data is converted into numerical data, which is convenient for input into machine learning models.

[0032] Outlier detection and replacement include: Outlier detection: Outliers may have a negative impact on model training, so they need to be detected and processed. In the embodiment of the present invention, a box plot method (BoxPlot) is used to detect outliers: First, calculate the quartiles of the data ( and ), and the interquartile range (IQR): , then, detect outliers using the following formula: in, and are the first and third quartiles, respectively, and IQR is the interquartile range; Outlier processing: For detected outliers, the median substitution method can be used to avoid excessively weakening the actual characteristics of the data: in, For the replaced data, is the original data.

[0033] Feature selection and dimensionality reduction: Feature selection: Feature selection is to select the features that have the greatest impact on the target variable from a large number of features. In the embodiment of the present invention, the Pearson correlation coefficient is used to select the features that are most relevant to the target variable (such as brain metastasis risk): ,in, and Characteristics and the target variable The value of and is their mean. Pearson correlation coefficient The value range of is [-1,1], where close to 1 or -1 indicates that the feature is highly correlated with the target variable; Dimensionality reduction: When the feature dimension is too high, principal component analysis (PCA) is used for dimensionality reduction. PCA reduces the number of features by mapping the data to a new feature space while trying to maintain the variance of the data: in, is the original data matrix, is the transformation matrix, is the data matrix after dimensionality reduction.

[0034] Furthermore, the lung CT image data is subjected to denoising, cropping, ROI extraction, resampling and enhancement processing. Specifically, denoising includes: Median Filtering: Used to remove salt and pepper noise while retaining edge information. Median filtering replaces pixel values ​​with the median of its neighborhood, which is suitable for removing isolated noise points in an image. The expression is: ,in, is the position in the original image The pixel value of For the location The pixel values ​​of the filtered image are It's pixels The surrounding neighborhood, for Any point in the neighborhood of ; Gaussian Filtering: Smooths the image and reduces Gaussian noise by weighted averaging the pixel values ​​in the area around the pixel. Gaussian filtering uses the following Gaussian kernel for convolution: in, is the standard deviation, which controls the smoothness of the filter. are pixel coordinates.

[0035] Cropping and ROI extraction: Image cropping: Remove irrelevant background areas in the image through cropping operations and focus on the lung area. The cropping area is determined by manual annotation or automated methods to ensure that the size and area of ​​the input image meet the analysis requirements; Lung region extraction: Use threshold segmentation methods (such as the Otsu algorithm) to automatically extract the lung region. The Otsu algorithm automatically selects the optimal threshold by minimizing the intra-class variance: in, is the probability density function of the gray level of the image, m represents the gray level of the image, and the Otsu algorithm selects the optimal threshold by minimizing the intra-class variance to distinguish lung tissue from non-lung tissue.

[0036] Resampling: Image resampling: Since the resolution of different CT scans may be different, the images need to be unified to the same resolution, usually 1mm³. Resampling uses interpolation methods such as CubicInterpolation to adjust: in, is the resampling mapping function, is the original CT image, is the resampled image.

[0037] Data Augmentation: Rotation: randomly rotate the image to enhance the adaptability to images at different angles; Translation: translate the image horizontally and vertically; Scaling: Randomly scale images to improve the model's adaptability to different scales; Intensity Adjustment: Simulate images under different scanning conditions by adjusting contrast and brightness.

[0038] Furthermore, the multi-parameter MRI image data of the brain are subjected to denoising, registration and resampling. Specifically, Gaussian filter is used to denoise the multi-parameter MRI image data of the brain, and the expression is: ,in, is the standard deviation of the Gaussian filter, which controls the smoothness of the filter; Align to a standard brain template: Use rigid registration to align brain MRI images with a standard brain template (such as the MNI template). Rigid registration methods include translation and rotation to ensure that brain images of different patients are aligned to a unified standard space; Resampling after registration: After registration, the resolution of the image may need to be adjusted. Use the CubicInterpolation method to resample the image to a standard resolution (e.g. 1mm³).

[0039] Step S02: perform anomaly detection on the preprocessed data to obtain a corresponding anomaly score.

[0040] Among them, the preprocessed clinical pathology data are detected for anomalies through the autoencoder, and the preprocessed lung CT image data and the preprocessed brain multi-parameter MRI image data are detected for anomalies through the Transformer autoencoder.

[0041] In this embodiment, an autoencoder is used to perform unsupervised anomaly detection on clinical pathology data to generate anomaly scores.

[0042] Clinical pathological features include but are not limited to the following: Numerical features: tumor size (such as maximum diameter, volume), patient age, etc.

[0043] Categorical features: such as TNM stage, gender, family history, smoking history, etc., have been converted into numerical form through one-hot encoding.

[0044] Input data format: Assume that the clinical pathological feature vector is , whose dimensions are ,in is the number of samples, is the number of features.

[0045] The autoencoder consists of an encoder and a decoder: Encoder: Transforms high-dimensional input features Compressing to a low-dimensional latent space .

[0046] Decoder: From latent space Reconstruct the original input features .

[0047] Encoder structure: Input layer: receiving dimensional feature vector .

[0048] Hidden layer: gradually reduce the feature dimension, using fully connected layers and activation functions (such as ReLU): in, is the encoder weight matrix, is the bias term, is the activation function.

[0049] Decoder structure: Input layer: receives latent space representation .

[0050] Hidden layer: gradually restore the feature dimension, using a fully connected layer and activation function: in, and are the weights and biases of the decoder respectively.

[0051] Loss function: The autoencoder learns the normal feature distribution by minimizing the reconstruction error between the input and the reconstructed data. The commonly used loss function is the mean square error (MSE): .

[0052] When training the model, clinical pathological data of normal patients (without brain metastasis) are used for training. This enables the model to accurately reconstruct the input features of normal samples while introducing significant reconstruction errors in abnormal samples (patients with high risk of brain metastasis). The reconstruction error (abnormal score) is calculated as follows: , calculate the reconstruction error ,The higher the reconstruction error, the more abnormal the clinical characteristics of the sample.

[0053] For the preprocessed lung CT image data and the preprocessed brain multi-parameter MRI image data, the Transformer Autoencoder is used for anomaly detection. The model is trained based on normal samples (non-metastatic patients), learns the characteristics of normal brain images and identifies abnormal areas. The Transformer Autoencoder also consists of an encoder and a decoder. By compressing the input data into a latent space and trying to recover the original data from it, the autoencoder can learn the key information in the data. The anomaly detection task utilizes the reconstruction error of the input data by the autoencoder, that is, when the difference between the input data and the reconstructed data is large, it indicates that the data is abnormal. By introducing the self-attention mechanism of the Transformer, the Transformer Autoencoder can capture longer-range dependencies in the encoding process of the data, which is particularly suitable for complex imaging data such as brain MRI images.

[0054] The task of the encoder part is to map the input brain multi-parameter MRI image to a low-dimensional latent space. The Transformer encoder uses the self-attention mechanism to capture long-term dependencies in image data, allowing the model to more accurately extract global features in the image. Its structure is as follows: 1.1 Input Layer The input data is a brain MRI image, assuming the image size is ,in is the height of the image, is the width, is the depth of the image, is the number of channels of the image (e.g., MRI data with multiple parameters may have multiple channels).

[0055] For the input image, we first perform image block embedding and cut the image into multiple image blocks (patches). The size of each block is usually or , each image patch is then flattened and mapped into a high-dimensional space.

[0056] 1.2. Patch Embedding The image is divided into multiple image blocks through slicing operations, each image block is flattened into a vector and mapped to a higher-dimensional space through linear transformation. The above steps are done through a linear transformation matrix Implemented: ,in, is the vector representation of the kth image block, Also described as the vector representation of the k-th region.

[0057] 1.3 Positional Encoding Since the Transformer model itself does not have position awareness, it is necessary to introduce positional encoding to retain the spatial position information of each image block. Positional encoding is usually a vector added to the input vector. ,in, is the position encoding vector, is the final vector after adding the position encoding.

[0058] 1.4. Self-Attention The core of Transformer is the self-attention mechanism, which allows each image block to interact with all other image blocks to capture global image information. For each image block, three vectors are first generated: query, key, and value. in, , , is the learned weight matrix, are query, key, and value vectors. By calculating the similarity between each query vector and the key vector, a weighted value vector is obtained to generate the relationship information between image patches.

[0059] 1.5. Multi-head Self-Attention In order to capture the various relationships between different parts of the image, a multi-head self-attention mechanism is used. By computing multiple self-attention heads in parallel, each head learns a different feature representation. The outputs of each head are then concatenated to form the final self-attention representation: in, represents the output of the h-th self-attention head, is the output transformation matrix.

[0060] 1.6. Feed-Forward Network The output of the Transformer encoder usually passes through a feedforward neural network, which performs a nonlinear transformation on the output of the self-attention. The network contains two fully connected layers and uses the ReLU activation function: in, is the weight matrix, is the bias term, is the input.

[0061] After multiple layers of self-attention and feed-forward neural networks, the encoder outputs a latent space representation is a compressed representation of the image. This latent vector contains the key information of the input image and will be fed into the decoder for reconstruction.

[0062] The task of the decoder is to transform the latent space representation of the encoder output Restore to the original image. The decoder usually contains multiple layers of deconvolution (transposed convolution) layers and upsampling layers to restore the image by gradually increasing the spatial resolution. Its structure is as follows: 2.1 Input Layer The input layer of the decoder is the latent space representation of the encoder output , which is a low-dimensional vector.

[0063] 2.2 Self-Attention Layer The decoder uses a self-attention mechanism to process the latent space representation , which helps the model capture the global dependencies of the image and further optimize the image reconstruction process. Its implementation method is the same as the encoder.

[0064] 2.3. Deconvolutional Layer In the decoder, deconvolution (also called transposed convolution) layers are used to reduce the latent space representation to higher dimensions, gradually recovering the spatial resolution of the image.

[0065] 2.4. Activation function: The last layer uses the Sigmoid activation function to limit the decoded image pixel values ​​to the range [0,1], which is suitable for restoring grayscale images: ; Final output of the decoder is the reconstructed image. The goal of the autoencoder is to optimize the model by minimizing the difference between the input image and the reconstructed image (reconstruction error). The commonly used loss function is the mean square error (MSE): in, is the sample size, is the input image, To reconstruct the image.

[0066] After training, the model is able to reconstruct normal brain MRI images. Anomaly detection is done by calculating the reconstruction error of the image (i.e., the difference between the input image and the reconstructed image, also known as the anomaly score). When the reconstruction error is large, it means that the image is significantly different from the pattern of normal data and is therefore marked as abnormal.

[0067] It can be understood that the expression of the anomaly score is:

[0068] in, represents the abnormal score of the i-th sample of the brain multi-parameter MRI image data, represents the input image of the i-th sample of brain multi-parameter MRI image data, represents the reconstructed image of the i-th sample of the brain multi-parameter MRI image data, represents the abnormal score of the i-th sample of lung CT image data, represents the input image of the i-th sample of lung CT image data, Represents the reconstructed image of the i-th sample of the lung CT image data.

[0069] Step S03: After weighting various types of anomaly scores, normalize them to obtain a comprehensive anomaly score.

[0070] Specifically, the weighted expressions of various anomaly scores are:

[0071] is the abnormal score of brain multi-parameter MRI image data, is the abnormal score of lung CT image data, is the abnormal score of clinical pathological data, , , They are the weight coefficients of the abnormal scores of brain multi-parameter MRI image data, lung CT image data, and clinical pathology data. The weight coefficients are usually selected based on the characteristics of the data or through cross-validation. In general, the abnormal scores of MRI and CT may have a greater influence, so they can be given a higher weight, while the weight of clinical pathology characteristics can be relatively small. The specific selection can be based on the actual situation of the data.

[0072] Normalize the weighted anomaly score to the range [0,1] for subsequent threshold selection. Normalization can use minimum-maximum normalization:

[0073] S combinedis the weighted result, min(S combined ) is the minimum value of the weighted result, max(S combined ) is the maximum value of the weighted result, S combined，norm is the comprehensive anomaly score, which indicates the final abnormality degree of each sample.

[0074] Step S04: determining a prediction threshold of the risk of brain metastasis of lung cancer based on the maximized Youden index.

[0075] It should be noted that by calculating the sensitivity and specificity under different thresholds, the threshold that maximizes the Youden index is selected as the final prediction threshold. , specifically, Youden's Index is defined as: in: Sensitivity: The proportion of positive samples correctly identified by the model as positive (i.e., patients who actually have brain metastases from lung cancer).

[0076] Specificity: The proportion of negative samples (i.e., patients without lung cancer brain metastases) that the model correctly identifies as negative.

[0077] Choose the optimal threshold by trying different thresholds To maximize the Youden index , select an optimal threshold. The calculation method is as follows: For each candidate threshold calculate: Sensitivity:

[0078] Specificity:

[0079] Among them, TP, TN, FP, and FN are the true positive, true negative, false positive, and false negative of the model prediction results, respectively.

[0080] Step S05: compare the comprehensive anomaly score with the prediction threshold to determine a prediction result.

[0081] Understandably, when , the patient was predicted to be at high risk of brain metastasis; , the patient was predicted to be at low risk of brain metastasis.

[0082] Step S06, when the prediction result shows that the user is a patient with brain metastasis of lung cancer, the preprocessed brain multi-parameter MRI image data is obtained, the preprocessed brain multi-parameter MRI image data is divided into a plurality of regions of preset sizes, and reconstruction errors are calculated respectively.

[0083] Specifically, first, the MRI image is segmented into multiple smaller image patches. These regions can be, for example, or The image is divided into patches. Each patch will be reconstructed separately. This patching method can effectively identify local high-risk areas.

[0084] For each region k, the reconstruction error is measured by calculating the difference between the input image and the reconstructed image. Reconstruction Error (Anomaly Score) The calculation formula is:

[0085] in, is the kth region of the original MRI image, is the kth region after reconstruction by the autoencoder, is the reconstruction error of the kth region.

[0086] Step S07, mapping the reconstruction errors of each region to generate an error map to reflect the high-risk regions of lung cancer brain metastasis.

[0087] The reconstruction error of each region Mapped onto the entire image, an error map is generated. Regions with large reconstruction errors indicate that the features of the region are significantly different from the normal distribution learned by the model, and are therefore considered potential high-risk regions. These regions may correspond to brain metastasis regions, and the model has not been exposed to the features of these regions during training, so it is difficult to reconstruct, resulting in large errors.

[0088] In some other embodiments of the present invention, in order to further improve the accuracy of high-risk areas, an enhancement score is determined according to the reconstruction error and the comprehensive anomaly score of each area. The expression of the enhancement score is:

[0089] in, Indicates the enhancement score, represents the abnormal score of the kth region of the brain multi-parameter MRI image data, represents the weighted result of the kth region, is the weight coefficient of the abnormal score of the kth region of the brain multi-parameter MRI image data, is the weight coefficient of the weighted result of the kth region; Determining whether the enhancement score is higher than a threshold; If it is determined that the enhancement score is higher than the threshold, the region where the enhancement score is higher than the threshold is determined as the final high-risk region for lung cancer brain metastasis.

[0090] It can be understood that the comprehensive anomaly score is combined with the reconstruction error, so that the area with a high comprehensive anomaly score and a high reconstruction error becomes a higher risk area. Finally, the calculated enhancement score is used to generate a heat map of the high-risk area. The heat map of the high-risk area can be displayed by color mapping (such as red indicates a high-risk area) to facilitate intuitive identification by doctors.

[0091] The heat map is superimposed on the original MRI image to show the high-risk areas. The transparency adjustment (alphablending) ensures that the heat map is integrated with the original image to enhance the focus on the high-risk areas.

[0092] For the identified high-risk areas, bounding boxes or other marking methods can be drawn on these areas to further emphasize the locations of these areas.

[0093] In summary, the lung cancer brain metastasis identification method and system based on multimodal data in the above-mentioned embodiments of the present invention obtains the user's clinical pathological data, lung CT image data and brain multi-parameter MRI image data, and performs data preprocessing respectively; performs abnormality detection on the preprocessed data to obtain the corresponding abnormality score; after weighting various abnormality scores, normalizes them to obtain a comprehensive abnormality score; determines the prediction threshold of the risk of lung cancer brain metastasis based on the maximized Youden index; compares the comprehensive abnormality score with the prediction threshold to determine the prediction result; when the prediction result shows that the user is a patient with lung cancer brain metastasis, the preprocessed brain multi-parameter MRI image data is divided into several regions of preset sizes, and reconstruction errors are calculated respectively; the reconstruction errors of each region are mapped to generate an error mapping diagram to reflect the high-risk areas of lung cancer brain metastasis.

[0094] Embodiment 2 See also Figure 2 , Figure 2 1 is a structural block diagram of a lung cancer brain metastasis recognition system based on multimodal data provided in Embodiment 2 of the present invention. The lung cancer brain metastasis recognition system based on multimodal data 200 includes: a preprocessing module 21, an abnormality detection module 22, a normalization processing module 23, a first determination module 24, a second determination module 25, a calculation module 26 and a mapping module 27, wherein: The preprocessing module 21 is used to obtain the user's clinical pathology data, lung CT image data and brain multi-parameter MRI image data, and perform data preprocessing respectively, wherein the clinical pathology data is subjected to data cleaning, feature encoding and conversion, outlier detection and replacement, feature selection and dimensionality reduction processing; the lung CT image data is subjected to denoising, cropping and ROI extraction, resampling and enhancement processing; the brain multi-parameter MRI image data is subjected to denoising, registration and resampling processing; The anomaly detection module 22 is used to perform anomaly detection on the preprocessed data to obtain a corresponding anomaly score, wherein the preprocessed clinical pathology data is detected by an autoencoder, and the preprocessed lung CT image data and the preprocessed brain multi-parameter MRI image data are detected by a Transformer autoencoder. Specifically, the expression of the anomaly score is:

[0095] in, represents the abnormal score of the i-th sample of the brain multi-parameter MRI image data, represents the input image of the i-th sample of brain multi-parameter MRI image data, represents the reconstructed image of the i-th sample of the brain multi-parameter MRI image data, represents the abnormal score of the i-th sample of lung CT image data, represents the input image of the i-th sample of lung CT image data, represents the reconstructed image of the i-th sample of lung CT image data, represents the abnormal score of the i-th sample of clinical pathology data, represents the input features of the i-th sample of clinical pathology data, Represents the reconstructed features of the i-th sample of clinical pathological data; The normalization processing module 23 is used to weight various anomaly scores and then perform normalization processing to obtain a comprehensive anomaly score. The expression of the weighted anomaly scores is:

[0096] The expression of the comprehensive anomaly score is:

[0097] in, , , are the weight coefficients of the abnormal scores of brain multi-parameter MRI image data, lung CT image data, and clinical pathology data, respectively. is the abnormal score of brain multi-parameter MRI image data, is the abnormal score of lung CT image data, is the abnormal score of clinical pathological data, is the weighted result, is the minimum value of the weighted result, is the maximum value of the weighted result, is the comprehensive anomaly score; A first determination module 24 is used to determine a prediction threshold of the risk of brain metastasis of lung cancer according to the maximized Youden index, wherein the threshold that maximizes the Youden index is selected as the final prediction threshold by calculating the sensitivity and specificity under different thresholds; A second determination module 25, configured to compare the comprehensive anomaly score with the prediction threshold to determine a prediction result; The calculation module 26 is used for obtaining the pre-processed brain multi-parameter MRI image data when the prediction result shows that the user is a patient with brain metastasis of lung cancer, dividing the pre-processed brain multi-parameter MRI image data into a plurality of regions of preset sizes, and calculating the reconstruction errors respectively; The mapping module 27 is used to map the reconstruction errors of each region to generate an error map to reflect the high-risk regions of lung cancer brain metastasis.

[0098] Furthermore, in some other embodiments of the present invention, the multimodal data-based lung cancer brain metastasis identification system 200 further includes: The third determination module is used to determine the enhancement score according to the reconstruction error and the comprehensive anomaly score of each region. The expression of the enhancement score is:

[0099] in, Indicates the enhancement score, represents the abnormal score of the kth region of the brain multi-parameter MRI image data, represents the weighted result of the kth region, is the weight coefficient of the abnormal score of the kth region of the brain multi-parameter MRI image data, is the weight coefficient of the weighted result of the kth region; A judging module, used to judge whether the enhancement score is higher than a threshold; The fourth determination module is used to determine the area with the enhancement score higher than the threshold as the final high-risk area for brain metastasis of lung cancer if it is determined that the enhancement score is higher than the threshold.

[0100] Embodiment 3 Another aspect of the present invention provides an electronic device, see Figure 3 , shown is an electronic device in Embodiment 3 of the present invention, including a memory 20, a processor 10, and a computer program 30 stored in the memory and executable on the processor. When the processor 10 executes the computer program 30, the method for identifying brain metastases of lung cancer based on multimodal data as described above is implemented.

[0101] In some embodiments, the processor 10 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor or other data processing chip, used to run program codes or process data stored in the memory 20, such as executing access restriction programs.

[0102] The memory 20 includes at least one type of readable storage medium, and the readable storage medium includes a flash memory, a hard disk, a multimedia card, a card-type memory (for example, an SD or DX memory, etc.), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the memory 20 may be an internal storage unit of an electronic device, such as a hard disk of the electronic device. In other embodiments, the memory 20 may also be an external storage device of an electronic device, such as a plug-in hard disk equipped on the electronic device, a smart memory card (SmartMediaCard, SMC), a secure digital (SecureDigital, SD) card, a flash card (FlashCard), etc. Further, the memory 20 may also include both an internal storage unit and an external storage device of the electronic device. The memory 20 may be used not only to store application software and various types of data of the electronic device, but also to temporarily store data that has been output or is to be output.

[0103] It should be pointed out that Figure 3 The structure shown does not constitute a limitation on the electronic device. In other embodiments, the electronic device may include fewer or more components than those shown in the figure, or combine certain components, or arrange the components differently.

[0104] The embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned method for identifying brain metastases of lung cancer based on multimodal data.

[0105] Those skilled in the art will appreciate that the logic and / or steps represented in the flowchart or otherwise described herein, for example, may be considered as an ordered list of executable instructions for implementing logical functions, and may be specifically implemented in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For purposes of this specification, "computer-readable medium" may be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.

[0106] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.

[0107] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or a combination thereof: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0108] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0109] The above embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present invention. It should be pointed out that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the present invention patent shall be subject to the attached claims.

Claims

1. A method for identifying brain metastases of lung cancer based on multimodal data, characterized in that: The method comprises: Obtain the user's clinical pathology data, lung CT image data, and brain multi-parameter MRI image data, and perform data preprocessing respectively; Perform anomaly detection on the preprocessed data to obtain the corresponding anomaly score, wherein the preprocessed clinical pathology data is detected by the autoencoder, and the preprocessed lung CT image data and the preprocessed brain multi-parameter MRI image data are detected by the Transformer autoencoder; After weighting various anomaly scores, normalization is performed to obtain a comprehensive anomaly score; Determine the prediction threshold of lung cancer brain metastasis risk based on maximizing the Youden index; Comparing the comprehensive anomaly score with the prediction threshold to determine a prediction result; When the prediction result shows that the user is a patient with brain metastasis of lung cancer, the preprocessed brain multi-parameter MRI image data is obtained, the preprocessed brain multi-parameter MRI image data is divided into a number of regions of preset sizes, and reconstruction errors are calculated respectively; The reconstruction errors of each region are mapped to generate an error map to reflect the high-risk areas of lung cancer brain metastasis.

2. The method for identifying brain metastases of lung cancer based on multimodal data according to claim 1, characterized in that: The step of mapping the reconstruction errors of each region to generate an error map to reflect the high-risk region of lung cancer brain metastasis includes: Determine the enhancement score based on the reconstruction error and comprehensive anomaly score of each region; Determining whether the enhancement score is higher than a threshold; If it is determined that the enhancement score is higher than the threshold, the region where the enhancement score is higher than the threshold is determined as the final high-risk region for lung cancer brain metastasis.

3. The method for identifying brain metastases of lung cancer based on multimodal data according to claim 2, characterized in that: In the data preprocessing step, the clinical pathology data is subjected to data cleaning, feature encoding and conversion, outlier detection and replacement, feature selection and dimensionality reduction; the lung CT image data is subjected to denoising, cropping and ROI extraction, resampling and enhancement; and the brain multi-parameter MRI image data is subjected to denoising, registration and resampling.

4. The method for identifying brain metastases of lung cancer based on multimodal data according to claim 3, characterized in that: In the step of performing anomaly detection on the preprocessed data to obtain a corresponding anomaly score, the expression of the anomaly score is: in, represents the abnormal score of the i-th sample of the brain multi-parameter MRI image data, represents the input image of the i-th sample of brain multi-parameter MRI image data, represents the reconstructed image of the i-th sample of the brain multi-parameter MRI image data, represents the abnormal score of the i-th sample of lung CT image data, represents the input image of the i-th sample of lung CT image data, represents the reconstructed image of the i-th sample of lung CT image data, represents the abnormal score of the i-th sample of clinical pathology data, represents the input features of the i-th sample of clinical pathology data, Represents the reconstructed features of the i-th sample of clinical pathological data.

5. The method for identifying brain metastases of lung cancer based on multimodal data according to claim 4, characterized in that: In the step of weighting various types of anomaly scores and then normalizing them to obtain a comprehensive anomaly score, the expression for weighting various types of anomaly scores is: The expression of the comprehensive anomaly score is: in, , , are the weight coefficients of the abnormal scores of brain multi-parameter MRI image data, lung CT image data, and clinical pathology data, respectively. is the abnormal score of brain multi-parameter MRI image data, is the abnormal score of lung CT image data, is the abnormal score of clinical pathological data, is the weighted result, is the minimum value of the weighted result, is the maximum value of the weighted result, is the comprehensive anomaly score.

6. The method for identifying brain metastases of lung cancer based on multimodal data according to claim 5, characterized in that: In the step of determining the prediction threshold of the risk of brain metastasis of lung cancer based on the maximized Youden Index, the sensitivity and specificity under different thresholds are calculated, and the threshold that maximizes the Youden Index is selected as the final prediction threshold.

7. The method for identifying brain metastases of lung cancer based on multimodal data according to claim 6, characterized in that: In the step of determining the enhancement score according to the reconstruction error and the comprehensive anomaly score of each region, the expression of the enhancement score is: in, Indicates the enhancement score, represents the abnormal score of the kth region of the brain multi-parameter MRI image data, represents the weighted result of the kth region, is the weight coefficient of the abnormal score of the kth region of the brain multi-parameter MRI image data, is the weight coefficient of the weighted result of the kth region.

8. A lung cancer brain metastasis recognition system based on multimodal data, characterized in that: A method for identifying brain metastases of lung cancer based on multimodal data according to any one of claims 1 to 7, the system comprising: The preprocessing module is used to obtain the user's clinical pathology data, lung CT image data, and brain multi-parameter MRI image data, and perform data preprocessing respectively; An anomaly detection module is used to perform anomaly detection on the preprocessed data and obtain the corresponding anomaly score. The preprocessed clinical pathology data is detected by an autoencoder, and the preprocessed lung CT image data and the preprocessed brain multi-parameter MRI image data are detected by a Transformer autoencoder. The normalization processing module is used to weight various anomaly scores and perform normalization processing to obtain a comprehensive anomaly score; A first determination module is used to determine a prediction threshold of the risk of brain metastasis of lung cancer based on maximizing the Youden index; A second determination module, configured to compare the comprehensive anomaly score with the prediction threshold to determine a prediction result; A calculation module, for obtaining pre-processed brain multi-parameter MRI image data when the prediction result shows that the user is a patient with brain metastasis of lung cancer, dividing the pre-processed brain multi-parameter MRI image data into a plurality of regions of preset sizes, and calculating reconstruction errors respectively; The mapping module is used to map the reconstruction errors of each region and generate an error map to reflect the high-risk areas of lung cancer brain metastasis.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method for identifying brain metastases of lung cancer based on multimodal data as described in any one of claims 1 to 7 is implemented.

10. An electronic device, characterized in that: The invention comprises a memory, a processor and a computer program stored in the memory and running on the processor, wherein when the processor executes the program, the method for identifying brain metastasis of lung cancer based on multimodal data according to any one of claims 1 to 7 is implemented.

Images