Pulmonary nodule benign and malignant detection method based on methylation sequence and medical image modal fusion

Abstract

The invention provides a pulmonary nodule benign and malignant detection method based on methylation sequence and medical image modal fusion, which comprises the following steps of: slicing three-dimensional image data, collecting nrrd format three-dimensional medical image data, slicing the three-dimensional data corresponding to each id into N parts, and only retaining the middle part of slices; extracting three-dimensional image data features, wherein one feature is a third-dimensional-based slice data group of original nrrd-format three-dimensional image data; the other one is a data group which is obtained by performing AND operation on mask data and nrrd format original data and mapping three-dimensional image data according to a third dimension as a basic slice; methylation sequence data enhancement is carried out, geometric transformation is carried out on a data set with mask data, then targeted cutting is carried out, sensitive parts are extracted in a targeted mode according to mask data, and noise addition is carried out before feature extraction; and on the basis of feature adaptive fusion of the existing data set, carrying out fusion processing on the feature vectors obtained through the above steps to obtain related feature vectors.

Description

Technical Field

[0001] The present invention relates to the technical field of lung nodule detection, and in particular provides a method for detecting the benign and malignant nature of lung nodules based on the fusion of methylation sequences and medical imaging modalities. Background Art

[0002] As an early manifestation of lung cancer, the accurate identification of the benign and malignant nature of lung nodules is of decisive significance for the selection of treatment plans and prognosis evaluation. With the continuous advancement of precision medicine and personalized treatment, the early diagnosis and treatment of lung cancer have become particularly important. The development of precision medicine requires us to be able to identify the benign and malignant nature of lung nodules earlier and more accurately in order to provide personalized treatment plans for patients.

[0003] Currently, the detection of lung nodules mainly relies on medical imaging, such as CT scans and MRI, as well as the detection of biomarkers, such as DNA methylation. Due to the complexity and diversity of lung nodules, a single detection method often fails to provide sufficient diagnostic information. Existing lung nodule detection methods, such as CT, MRI, and DNA methylation detection, although they have improved the diagnostic accuracy to a certain extent, there are still risks of misdiagnosis and missed diagnosis. For example, imaging methods may be affected by factors such as image quality and operator experience, while biomarker detection may be limited by sample quality and detection technology. Therefore, there are certain limitations in single-modal detection methods, and a single medical imaging or biomarker detection method may not be able to comprehensively reflect the characteristics of lung nodules, resulting in diagnostic uncertainty.

[0004] Theoretically, fusing different detection methods in a multi-modal manner can improve the diagnostic accuracy. However, how to effectively integrate and analyze these data from different sources is a technical challenge. In actual clinical applications, the interpretability, computational efficiency, and clinical applicability of the algorithm need to be considered to ensure the accuracy and reliability of the diagnosis. Different-modal data have different data structures and characteristics, and how to effectively fuse these data and extract useful information is a technical problem. Existing multi-modal fusion algorithms often require complex parameter adjustment and optimization, which increases the difficulty of clinical application. Summary of the Invention

[0005] To solve the above problems, the present invention proposes a method for detecting the benign and malignant nature of lung nodules based on the fusion of methylation sequences and medical imaging modalities, so as to utilize deep learning technology, especially an improved adaptive modal fusion algorithm, to integrate methylation sequence features and three-dimensional medical imaging data and improve the accuracy of detecting the benign and malignant nature of lung nodules.

[0006] To achieve the above object, the present invention provides a method for detecting the benign and malignant nature of lung nodules based on the fusion of methylation sequences and medical imaging modalities, including:

[0007] 3D image data slicing: Collect 3D medical image data in nrrd format, slice the 3D data corresponding to each ID into N parts, and only retain the middle part of the slices;

[0008] Extracting 3D image data features: One is a slice data group of the original nrrd format 3D image data based on the third dimension; the other is a data group of the mapped 3D image data obtained by performing an AND operation on the mask mask data and the original nrrd format data, sliced based on the third dimension;

[0009] Performing methylation sequence data enhancement: Geometric transformation is performed on the dataset with mask mask data, and then targeted cropping is carried out. The sensitive part is extracted according to the mask data, and noise is added before feature extraction;

[0010] Feature adaptive fusion based on the existing dataset: The feature vectors obtained through the above steps are fused to obtain relevant feature vectors.

[0011] Furthermore, the process of feature adaptive fusion based on the existing dataset includes:

[0012] First, after slicing the original 3D image in nrrd format, within the group of feature vectors corresponding to a set of extracted IDs, every twelve intra-group vectors form a small group and are fused with the corresponding methylation feature vectors to obtain a small group fusion feature vector corresponding to one ID; then, the obtained multiple small group fusion feature vectors are further subjected to multi-modal feature fusion to obtain the final feature vector.

[0013] Furthermore, in the process of feature adaptive fusion based on the existing dataset, the method of low-rank multi-modal fusion is used, and effective multi-modal fusion is carried out using low-rank factors, including:

[0014] First, initialize a set of low-rank factors for each group of feature vectors, and the dimension of the low-rank factors is the product of the dimension of each modal vector and a preset low rank r; then, each group performs parallel decomposition, and the representation of each modality is decomposed in parallel to calculate the product of its low-rank factor and the modal vector for each modality: Z*W = h; finally, perform an element-wise product on the h obtained by decomposing each group, perform an element-wise product on the decomposition results of all modalities, and the total feature is obtained after fusion.

[0015] The method of low-rank multi-modal fusion is used for multi-modal data fusion, effectively solving the problem of too high dimensionality of feature vectors and ensuring the quality of the fused features and the accuracy of detection.

[0016] Furthermore, the three groups of feature vectors for feature adaptive fusion based on the existing dataset include: the feature vectors extracted from the two-dimensional data sliced from the nrrd-format three-dimensional medical image data with the patient ID as the grouping premise; the feature vectors obtained by slicing the three-dimensional mask image data and the three-dimensional original nrrd-format medical image data after AND operation mapping into two dimensions, grouping them with the patient ID as the premise, and using the VIT model for extraction; and the methylation sequence feature vectors grouped with the patient ID as the premise.

[0017] Furthermore, after collecting the nrrd-format medical image data, perform AND operation mapping processing on the sensitive areas of the original data, including:

[0018] When there is corresponding mask data, first perform mapping on the original nrrd-format image data and the mask sensitive mask data, perform AND operation on the nrrd-format original image data and the corresponding mask data, and only retain the part of the nrrd format that the mask is interested in to obtain targeted three-dimensional data;

[0019] When there is no mask data, there is no need to perform mapping and operation processing on the original nrrd-format data, and directly proceed to the next preprocessing step.

[0020] Furthermore, during the slicing process of the three-dimensional medical image data, the three-dimensional data for slicing processing includes three-dimensional mask data, three-dimensional nrrd-format data, and three-dimensional data after AND operation mapping.

[0021] Furthermore, use the VIT model to extract the features of the three-dimensional medical image data. The process includes: deploying the VIT model, loading the pre-trained VIT model, performing image preprocessing, and extracting feature vectors. Utilizing deep learning techniques, especially the pre-trained VIT model, the present invention can automatically extract and learn complex features in medical images and methylation sequences, reducing the dependence on professional knowledge and experience.

[0022] Furthermore, before data augmentation of the methylation sequence data, perform preprocessing of the methylation dataset, read the methylation data, use the file ID as the label, store it in an ID list, and each element of the list serves as a key, and the value corresponding to the key is a series of chromosome methylation sequences pointed to after the corresponding ID. Store the benign and malignant states in the list to represent whether the corresponding sequence is benign or malignant.

[0023] Furthermore, during the data augmentation process of the methylation sequence data, each time a group of slice image data corresponding to an ID is imported, randomly add Gaussian noise to 60% of the data as new data for feature extraction. Through data augmentation techniques such as targeted cropping and noise addition, the present invention can improve the generalization ability of the model for small-sample data, increase the data volume, and improve the robustness of the model.

[0024] Furthermore, during the methylation sequence data enhancement process, targeted cropping is performed: according to the mask image mask data, the sensitive part of the methylation sequence data is extracted targeted, and it is cropped into three parts centered on the sensitive area with values in the mask, with a size of 1 / 2 - 1 / 4 of the original image. The part outside the boundary of 512x512 after cropping is filled with 0.

[0025] Through data enhancement techniques such as targeted cropping and noise addition, the present invention can improve the generalization ability of the model for small sample data, increase the data volume, and improve the robustness of the model.

[0026] Compared with the traditional single-modal prediction method, the present invention has the following technical advantages:

[0027] 1. Multi-modal data fusion: By combining methylation sequences and three-dimensional medical image data, the present invention can utilize the complementary information in the two modalities, improving the accuracy and reliability of lung nodule benign and malignant detection.

[0028] 2. Adaptive modal fusion algorithm: The improved adaptive modal fusion algorithm can better adapt to the characteristics of different data modalities, solve the limitations that may exist in single-modal data, and improve the comprehensiveness and accuracy of detection.

[0029] 3. Deep learning technology: Using deep learning technology, especially the pre-trained VIT model, the present invention can automatically extract and learn complex features in medical images and methylation sequences, reducing the dependence on professional knowledge and experience.

[0030] 4. Data enhancement technology: Through data enhancement techniques such as targeted cropping and noise addition, the present invention can improve the generalization ability of the model for small sample data, increase the data volume, and improve the robustness of the model.

[0031] 5. Low-rank multi-modal fusion: The low-rank multi-modal fusion (LMF, Low-rank Multimodal Fusion) method is used for multi-modal data fusion, effectively solving the problem of too high dimensionality of feature vectors, and ensuring the quality of fused features and the accuracy of detection.

[0032] 6. Compared with single-modal methods, the accuracy of the fusion modality is significantly improved under different models. For example, the accuracies of the Res-net, CNN, and Transformer models under the fusion modality are 81.2%, 71.7%, and 82.3% respectively, which are significantly better than the performance of single-modal data. By integrating multiple data sources, it helps to reduce misdiagnosis and missed diagnosis caused by the limitations of a single data source, improves the accuracy of diagnosis, reduces misdiagnosis and missed diagnosis, helps clinicians identify the benign and malignant nature of pulmonary nodules earlier and more accurately, provides personalized treatment plans for patients, and has significant clinical application potential. Brief Description of the Drawings

[0033] The drawings described herein are used to provide a further understanding of the present invention, form a part of this application, and the schematic embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0034] Figure 1 It is a flow chart of the sensitive area and operation mapping processing provided;

[0035] Figure 2 It is a flow chart of the three-dimensional image data slicing provided;

[0036] Figure 3 It is a flow chart of the VIT extracting image features provided;

[0037] Figure 4 It is a schematic diagram of the feature vector acting on the low-rank factor provided;

[0038] Figure 5 It is a flow chart of the multi-modal fusion provided. Detailed Description of the Invention

[0039] Deep learning technology has made remarkable progress in the fields of image recognition and bioinformatics, providing new possibilities for the detection of the benign and malignant nature of pulmonary nodules. Multi-modal fusion enables us to utilize the complementary information existing in multi-modal data, thereby discovering the dependence of information on multi-modalities.

[0040] Currently, the detection of pulmonary nodules mainly relies on medical imaging, such as CT scans and MRI, as well as the detection of biomarkers, such as DNA methylation. Due to the complexity and diversity of pulmonary nodules, a single detection method often fails to provide sufficient diagnostic information, resulting in diagnostic uncertainty.

[0041] In theory, multimodal fusion of different detection methods can improve the accuracy of diagnosis. However, how to effectively integrate and analyze data from different sources is a technical challenge. In actual clinical applications, the interpretability, computational efficiency, and clinical applicability of the algorithm need to be considered to ensure the accuracy and reliability of diagnosis. Different modalities of data have different data structures and characteristics. How to effectively fuse this data and extract useful information is a technical problem. Existing multimodal fusion algorithms often require complex parameter adjustment and optimization, which increases the difficulty of clinical application.

[0042] To this end, the present invention provides a method for detecting the benign and malignant nature of pulmonary nodules based on the fusion of methylation sequences and medical image modalities, including:

[0043] Three-dimensional image data slicing: Collect three-dimensional medical image data in nrrd format, slice the three-dimensional data corresponding to each id into N parts, and only retain the middle part of the slices;

[0044] Extracting three-dimensional image data features: One is a slice data group based on the third dimension of the original three-dimensional image data in nrrd format; the other is a data group of mapped three-dimensional image data obtained by performing an AND operation on the mask mask data and the original data in nrrd format and sliced based on the third dimension;

[0045] Performing methylation sequence data augmentation: Geometric transformation is performed on the data set with mask mask data, and then targeted cropping is carried out. The sensitive part is extracted according to the mask data, and noise is added before feature extraction;

[0046] Feature adaptive fusion based on existing data sets: The feature vectors obtained through the above steps are fused to obtain relevant feature vectors.

[0047] The method for detecting the benign and malignant nature of pulmonary nodules based on the fusion of methylation sequences and medical image modalities provided by the present invention is an innovation regarding an adaptive modality fusion algorithm, which is specifically for methylation sequences and three-dimensional medical image data in nrrd format. The core of the present invention lies in combining methylation sequence features with three-dimensional medical image data to achieve efficient detection of the benign and malignant nature of pulmonary nodules.

[0048] Among them, the process of feature adaptive fusion based on existing data sets includes:

[0049] First, after slicing the original three-dimensional image in nrrd format, within the group of feature vectors corresponding to a set of extracted ids, every twelve in-group vectors form a small group and are fused with the corresponding methylation feature vectors to obtain a small group of fusion feature vectors corresponding to one id;

[0050] Then, the obtained multiple group fusion feature vectors are further subjected to multi-modal feature fusion to obtain the final feature vector.

[0051] Moreover, in the process of feature adaptive fusion based on the existing data set, a low-rank multi-modal fusion method is used, and effective multi-modal fusion is carried out by using low-rank factors, including:

[0052] First, initialize a set of low-rank factors for each group of feature vectors, and the dimension of the low-rank factors is the product of the dimension of each modal vector and a preset low rank r;

[0053] Then, each group performs parallel decomposition, and the representation of each modality is decomposed in parallel to calculate the product of its low-rank factor and the modal vector for each modality: Z*W = h;

[0054] Finally, perform an element-wise product on the h obtained by decomposing each group, perform an element-wise product on the decomposition results of all modalities, and the total feature is obtained after fusion.

[0055] Through this advanced fusion strategy, the present invention can make full use of the complementary information of the two-modal data, thereby improving the accuracy and reliability of lung nodule detection. This improved algorithm not only optimizes the data processing flow, but also enhances the model's ability to analyze complex medical image data, providing an accurate auxiliary diagnosis tool for clinicians.

[0056] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. The following description of at least one exemplary embodiment is actually only illustrative and in no way constitutes a limitation on the present invention and its application or use. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present invention. The specific implementation process is as follows:

[0057] (1) Processing of methylation data set

[0058] As Figure 1 shown, to better preprocess the methylation sequence, taking a methylation feature as an example, the specific operation steps are introduced here. Generally, a specific methylation data includes the following content:

[0059] id: The unique identifier of each sample;

[0060] Chromosome position: Indicates the chromosome position where the methylation site is located, in the format of chrX:XXXXX-XXXXX, where chrX represents chromosome x, and XXXXX-XXXXX represents the start and end positions of the site;

[0061] Methylation value: The methylation level at each locus, usually represented by a β-value, ranging from 0 (unmethylated) to 1 (fully methylated). These values can also be expressed as percentages, e.g., 75% indicates that 75% of the cytosines at that locus are methylated.

[0062] Malignancy classification: Represents the classification of the malignancy degree of the sample, which is divided into Malignant and Benign here.

[0063] Based on these characteristics of the methylation sequence, this method uses Python to read the methylation data, takes the file ID as the label, stores it in an ID list, and each element of this list serves as a key, with the value corresponding to the key being a series of chromosomal methylation sequences pointed to after the corresponding ID. Finally, the Malignancy classification is used as the labels and stored in the list to characterize whether the sequence is benign or malignant, facilitating subsequent training and fusion.

[0064] (2) Processing of the nrrd - format three - dimensional medical image dataset

[0065] Processing of the sensitive region of the original data through AND operation mapping. This step is divided according to whether there is corresponding mask - sensitive data.

[0066] With corresponding mask - sensitive data: First, map the original nrrd - format image data and the mask - sensitive mask data. Since the mask data is only valid in the region of interest and the rest is 0, this method directly performs a point - by - point AND operation on the original nrrd - format image data and the corresponding mask data, that is, only the part of the nrrd format that the mask is interested in is retained. After this step, a targeted three - dimensional data after the AND operation is obtained.

[0067] Without mask - sensitive data: The dataset without mask data does not need to perform mapping and operation processing on the original nrrd - format data, and can directly proceed to the next pre - processing step.

[0068] (3) Slicing of three - dimensional image data

[0069] Such as Figure 2As shown, since the nrrd - format medical image data is three - dimensional and the size of the third dimension is variable, the data used in this method is 512x512xN (where N ranges from 100 to 400). This three - dimensional variable data poses a relatively large obstacle to our subsequent work of extracting feature vectors. Therefore, this method slices each three - dimensional image corresponding to each id by N, splitting the data of one id into N parts. Visualizing these N parts of data, it can be found that the beginning and ending parts of the N parts are all completely black, with basically no useful information. So, this method discards the first 10 slices and the last 10 slices, reducing both the computational amount and interference. (The slicing operation in this step is performed on all three - dimensional data in the dataset, including the three - dimensional mask data (if any), the three - dimensional nrrd - format data, and the three - dimensional data after the AND operation mapping of the mask and nrrd - format data (if any).

[0070] (4) Extracting features of three - dimensional image data using the VIT model

[0071] As Figure 3 shown, deploy VIT, install the required libraries and dependency environments using pip (pip install torchtorchvision transformers pillow), and then import the PyTorch library, the Transformer library, and the vit library.

[0072] Load the pre - trained VIT model: The pre - trained model can be obtained from the Transformers library of Hugging Face or other resources. In this method, the pre - trained VIT model is obtained from Hugging Face for image feature extraction.

[0073] Image pre - processing: Convert the image into the format required by the model, including resizing, normalization, etc. The method of pre - processing has been mentioned in the second step of this method (slicing three - dimensional data, performing mapping operations, etc.). Currently, our data is 512x512.

[0074] Feature extraction: Input the pre - processed image into the model to obtain the feature vector. This step is relatively important. There are two types of data that need to be extracted in this method. One is the slice data group of the original three - dimensional nrrd - format image data based on the third dimension; the other is the slice data group of the three - dimensional image data after the AND operation mapping of the mask data and the original nrrd - format data, sliced based on the third dimension (this slicing is performed according to the method mentioned in the second step).

[0075] (5) Data augmentation

[0076] There are about a hundred methylation sequence data used in this method, and the data volume is relatively small. If there is enough data volume, this step can be omitted.

[0077] Geometric transformation (this method is only applicable to datasets with mask data):

[0078] Targeted cropping: According to the mask data, the sensitive part is extracted specifically, and it is cropped into three parts centered on the mask with a size of 1 / 2 - 1 / 4 of the original image. After cropping, the part outside the 512x512 boundary is filled with 0. In this way, we can obtain four groups of 512x512 data for feature extraction, and all four groups of data retain the feature information we need.

[0079] Noise addition: For some previous datasets, noise is added before feature extraction.

[0080] Each time a set of slice image data corresponding to an id is imported for feature extraction, Gaussian noise is randomly added to 60% of the data as new data for feature extraction. In this way, the previous data volume can be expanded by 60% and it is more robust. Of course, this 60% should not be set too large, as it may cause situations such as non-convergence and overfitting in subsequent training.

[0081] (VI) Multi-modal data fusion

[0082] After the first four steps, three groups of feature vectors are obtained, specifically including: 1. Feature vectors extracted from two-dimensional data sliced from three-dimensional image data in nrrd format based on patient id as the grouping premise; 2. Feature vectors obtained by slicing two-dimensional data after the operation mapping of mask three-dimensional image data and nrrd format three-dimensional original image data, grouping by patient id, and using the VIT model for extraction; 3. Methylation sequence feature vectors grouped by patient id.

[0083] Since the above three types of feature vectors are grouped based on patient id, and the first two are data groups sliced from three-dimensional image data (512x512xN, N is generally 100 - 400), and generally there are hundreds of feature vectors extracted from this data group, while each id of the methylation feature vector only corresponds to one feature vector. Therefore, directly fusing them will undoubtedly result in the phenomenon of large swallowing small (that is, the feature vectors of the image grouping are too large, while the proportion of the methylation sequence vector is very small. Therefore, the features on the methylation sequence side will not be well retained and will be swallowed by the large image grouping feature vectors).

[0084] This method proposes a feature adaptive fusion strategy based on an existing dataset, which solves the above problems and improves the prediction accuracy to a certain extent compared with single-modal prediction.

[0085] The specific fusion implementation method is as follows:

[0086] (1) For a group of feature vectors corresponding to a set of IDs extracted after slicing the original 3D image in nrrd format, every twelve intra-group vectors within the group are taken as a small group, which is fused with the corresponding methylation feature vector to obtain a fused feature vector of a small group corresponding to an ID. Since there are multiple small groups for the 3D slices corresponding to a set of IDs, each small group needs to be fused with the corresponding methylation feature vector to obtain the fused feature vectors of multiple small groups.

[0087] (2) The fused feature vectors of these multiple small groups are then subjected to multi-modal feature fusion to obtain the final feature vector. This feature vector not only retains the features of each data group but is not overly numerous, well meeting the requirements of subsequent operations.

[0088] (3) Due to the excessive number of groups, this undoubtedly becomes a multi-dimensional fusion, far greater than 3D. Then there are also requirements for the fusion method used. The traditional TFN fusion uses the Cartesian inner product of each modality to be fused. The advantage of this is that all features of all modalities are retained, but the disadvantage is that the size of the fused feature vector will increase exponentially with the fusion dimension, which is definitely not suitable for this method. Since there are dozens or hundreds of feature vectors to be fused corresponding to each group of IDs, and their dimension is very high, this method adopts the LMF (Low-rank Multimodal Fusion) low-rank multi-modal fusion method, using low-rank weight tensors for effective multi-modal fusion.

[0089] (4) First, initialize a set of low-rank factors for each group of feature vectors. The dimension of these factors should be the product of the dimension of each modality vector and a preset low rank r. For example, if the dimension of a modality is dm, then the low-rank factor of each modality can be a dm×r matrix;

[0090] (5) Then each group is decomposed in parallel, and the representation of each modality is decomposed in parallel. It is necessary to calculate the product of the low-rank factor and the modality vector for each modality. As shown in the following figure, Z*W = h (where Z is the modality feature, W is the minimum-rank vector matrix, and h is the output feature); (obtain a 1xh-dimensional feature. In the figure, Zv can be regarded as the methylation feature vector, and Zl can be regarded as the feature vector of the slice group image data). (As Figure 4 shown)

[0091] (6) Then perform an element-wise product on the h obtained from each group: Perform an element-wise product (Hadamard product) on the decomposition results of all modalities. This step multiplies the low-rank representations of all modalities, rather than performing matrix multiplication. The element-wise product H = °°.........° (represents the ij-th element in the x-th decomposition result of each group, and the element-wise product is the multiplication of corresponding elements.) H is the total feature obtained after fusion.

[0092] By combining methylation sequences and three-dimensional medical image data, the present invention can utilize the complementary information in the two modalities, improving the accuracy and reliability of the detection of the benign and malignant nature of lung nodules. The improved adaptive modality fusion algorithm can better adapt to the characteristics of different data modalities, solve the limitations that may exist in single-modal data, and improve the comprehensiveness and accuracy of the detection.

[0093] (VII) Input for model training and verification

[0094] This application mainly provides an adaptive modality fusion strategy to improve the accuracy of predicting the benign and malignant nature of tumors, early screening of diseases, etc. using traditional medical image single modalities as training data. The specific model used is determined according to the actual problem faced.

[0095] Based on the data mentioned in this method, use the finally obtained fusion features with multiple models to predict and compare with the features generated by a single image, and obtain the following results.

[0096] Comparison of the accuracy rates of the single modality and the provided fusion modality

[0097]

[0098]

[0099] From the data in the table, it can be seen that the accuracy rate of the fused modality is improved compared with the single-modal data under different models). Compared with the single-modal method, the accuracy rate of the fused modality is significantly improved under different models. For example, the accuracies of the Res-net, CNN, and Transformer models in the fused modality are 81.2%, 71.7%, and 82.3% respectively, which are significantly better than the performance of single-modal data. By integrating multiple data sources, it helps to reduce misdiagnosis and missed diagnosis caused by the limitations of a single data source, improves the accuracy of diagnosis, reduces misdiagnosis and missed diagnosis, helps clinicians identify the benign and malignant nature of lung nodules earlier and more accurately, and provides personalized treatment plans for patients, showing significant clinical application potential.

[0100] Through deep learning techniques, especially the improved adaptive modality fusion algorithm, integrating methylation sequence features and three-dimensional medical image data, the multi-modal fusion and classification model can better adapt to the particularities of three-dimensional medical image data and methylation sequences, so as to improve the efficiency and accuracy of benign and malignant detection of lung nodules, reduce the dependence on professional knowledge and experience, and provide a powerful auxiliary diagnosis tool for clinicians.

[0101] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them; although the present invention has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that it is still possible to modify the specific implementation manners of the present invention or perform equivalent replacements for some technical features; without departing from the spirit of the technical solutions of the present invention, they should all be covered within the scope of the technical solutions claimed by the present invention.

Claims

1. A method for detecting benign and malignant pulmonary nodules based on methylation sequence and medical imaging modality fusion, characterized in that: include: 3D image data slicing: Collect 3D medical image data in nrrd format, slice the 3D data corresponding to each id into N pieces, and only keep the middle slices; Extracting 3D image data features: one is a slice data set based on the third dimension of the original nrrd format 3D image data; the other is a slice data set based on the third dimension of the mapped 3D image data obtained by performing AND operation on the mask data and the original nrrd format data; Perform methylation sequence data enhancement: geometrically transform the dataset with masked data, then perform targeted cropping, extract sensitive parts based on the masked data, and add noise before extracting features; Adaptive feature fusion based on existing data sets: The feature vectors obtained through the above steps are fused to obtain relevant feature vectors.

2. The method for detecting benign and malignant pulmonary nodules based on methylation sequence and medical imaging modality fusion according to claim 1, characterized in that: The process of adaptive fusion of features based on existing data sets includes: First, after the original 3D image in nrrd format is sliced, in the feature vector group corresponding to a set of extracted ids, every twelve vectors in the group are fused with the corresponding methylation feature vector to obtain a group fusion feature vector corresponding to the id; Then, the obtained multiple group fusion feature vectors are subjected to multimodal feature fusion to obtain the final feature vector.

3. The method for detecting benign and malignant pulmonary nodules based on methylation sequence and medical imaging modality fusion according to claim 2, characterized in that: In the process of feature adaptive fusion based on existing data sets, the low-rank multimodal fusion method is used to utilize low-rank factors for effective multimodal fusion, including: First, each group of feature vectors is initialized with a set of low-rank factors, where the dimension of the low-rank factor is the product of the dimension of each modal vector and a preset low-rank r; Then, each group is decomposed in parallel, and the representation of each mode is decomposed in parallel, and the product of its low-rank factor and the modal vector is calculated for each mode: Z*W = h (where Z is the modal feature, W is the minimum rank vector matrix, and h is the output feature); Finally, the element-wise product of h obtained from each group of decomposition is performed, and the element-wise product of the decomposition results of all modes is performed, and the total features are obtained after fusion.

4. The method for detecting benign and malignant pulmonary nodules based on methylation sequence and medical imaging modality fusion according to claim 1, characterized in that: The three groups of feature vectors based on feature adaptive fusion of existing data sets include: feature vectors extracted after slicing the three-dimensional image data in nrrd format into two-dimensional data based on the patient ID as the premise of grouping; mask three-dimensional image data and three-dimensional original image data in nrrd format are sliced ​​into two dimensions after operation mapping, and grouped based on the patient ID and extracted using the VIT model to obtain feature vectors; methylation sequence feature vectors obtained by grouping based on the patient ID.

5. The method for detecting benign and malignant pulmonary nodules based on methylation sequence and medical imaging modality fusion according to claim 1 or 4, characterized in that: After collecting medical imaging data in nrrd format, the sensitive areas of the original data are mapped and processed, including: When there is corresponding mask data, first use the original nrrd format image data and mask-sensitive mask data for mapping, perform point AND operation on the original nrrd format image data and the corresponding mask data, and only retain the part of the mask that is of interest to the nrrd format to obtain targeted three-dimensional data; When there is no mask data, there is no need to map and process the original data in nrrd format, and the next step of preprocessing can be performed directly.

6. The method for detecting benign and malignant pulmonary nodules based on methylation sequence and medical imaging modality fusion according to claim 1, characterized in that: In the process of slicing three-dimensional image data, the three-dimensional data processed by slicing includes mask three-dimensional data, nrrd format three-dimensional data and three-dimensional data after operation mapping.

7. The method for detecting benign and malignant pulmonary nodules based on methylation sequence and medical imaging modality fusion according to claim 1, characterized in that: The VIT model is used to extract the features of 3D image data. The process includes: deploying VIT, loading the pre-trained VIT model, performing image preprocessing, and extracting feature vectors.

8. The method for detecting benign and malignant pulmonary nodules based on methylation sequence and medical imaging modality fusion according to claim 1, characterized in that: Before methylation sequence data enhancement, the methylation data set is preprocessed to read the methylation data and store it in an ID list with the file ID as the label. Each element of the list is a key, and the value corresponding to the key is a series of chromosome methylation sequences pointed to by the corresponding ID. The benign and malignant are stored in the list to characterize the corresponding sequence as benign or malignant.

9. The method for detecting benign and malignant pulmonary nodules based on methylation sequence and medical imaging modality fusion according to claim 1 or 8, characterized in that: In the process of methylation sequence data enhancement, each time a set of slice image data corresponding to an ID is imported, Gaussian noise is randomly added to 60% of the data as new data extraction features.

10. The method for detecting benign and malignant pulmonary nodules based on methylation sequence and medical imaging modality fusion according to claim 9, characterized in that: During the methylation sequence data enhancement process, targeted cropping is performed: based on the mask data, the sensitive part of the methylation sequence data is extracted in a targeted manner, and the image is cropped into three parts with the mask as the center, at a size of 1 / 2-1 / 4 of the original image. After cropping, the part outside the boundary of 512x512 is filled with 0.

Images