Intelligent pulmonary nodule grading method and system based on multi-modality feature fusion

Abstract

An intelligent pulmonary nodule grading method and system based on multi-modality feature fusion. The method comprises: using clinical multi-modality data of a physical examination crowd to acquire an ROI and a VOI of a pulmonary nodule on the basis of a chest CT examination image and an examination report, designing a multi-task feature extraction network based on an attention mechanism, and acquiring radiomics features and deep-layer image features from the ROI and the VOI; designing a cross-modality feature fusion method based on graph representation learning, designing a multi-modality information extraction method, acquiring a specific feature representation and graph structure of each modality, and then performing feature representation and graph structure fusion; and providing a pulmonary nodule grading GCN model optimization and clinical verification method based on self-supervised learning, so as to implement fine grading of pulmonary nodule malignancy having only slight differences. A new way is provided for the design of a fine-grained classification algorithm; and in the aspect of solving a clinical problem, provided is a classification algorithm optimization method and clinical verification scheme in which generative self-supervised learning and comparative self-supervised learning are organically combined.

Description

Intelligent grading method and system for pulmonary nodules based on multimodal feature fusion

[0001] This application claims priority to the Chinese patent application filed with the Patent Office of China on July 26, 2023, with application number 202310927581.2 and invention name “Intelligent grading method and system for pulmonary nodules based on multimodal feature fusion”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present invention relates to a method and system for intelligent grading of pulmonary nodules based on multimodal feature fusion. Background Art

[0003] Low-dose computed tomography (LDCT) is an important means of detecting early lung cancer and can reduce the mortality rate of lung cancer by 20%. However, with the ultra-high growth of medical imaging data, there is a serious shortage of radiologists. Relying solely on radiologists for diagnosis is prone to missed or misdiagnosed cases due to subjective factors such as fatigue. According to statistics, approximately 30% of lung nodules removed during early treatment surgery are benign nodules, which means that for every lung cancer death prevented, 1.38 cases of overdiagnosis may occur, which not only increases the mental and economic burden on patients, but also causes a huge waste of social medical resources. Therefore, how to correctly assess the malignancy of lung nodules found in LDCT screening and assist doctors in taking correct clinical management measures has become an important clinical problem that needs to be urgently solved in lung cancer screening.

[0004] Lung-RADS (lung imaging reporting and data system) is a low-dose CT scan lung cancer screening classification standard published by the American College of Radiology (ACR). It aims to correctly classify, describe, and manage lung nodules, standardize follow-up and management decisions, avoid unnecessary repeated examinations and inappropriate interventions, and minimize patient anxiety and physician confusion. Lung-RADS categorizes lung nodules into four grades based on nodule type, size, and growth pattern, from low to high malignancy: Grades 1 and 2 are benign, requiring annual CT scans; Grade 3 is low-risk, requiring CT scans every six months; Grade 4 is further divided into three levels: 4A is intermediate-risk, requiring CT scans every three months; and 4B and 4X are high-risk nodules, requiring further detailed examination or surgery. Although doctors can reduce the false positives of LDCT screening to a certain extent by making clinical diagnosis of lung nodules according to the Lung-RADS grading standards, the lack of intelligent diagnostic tools corresponding to the Lung-RADS grading standards results in low sensitivity in clinical application, which may lead to delayed diagnosis of malignant lung nodules.

[0005] A study and cost analysis of lung nodule management strategies for baseline lung cancer screening in the United States showed that combining Lung-RADS grading with artificial intelligence (AI) can reduce the number of follow-up examinations and significantly save costs. Therefore, it is essential to develop Lung-RADS grading standards suitable for Asian populations and a corresponding intelligent grading tool for lung nodule malignancy to promote standardized clinical diagnosis and treatment of lung nodules.

[0006] With the rapid development of AI technology in recent years, AI technology not only plays an important role in the automatic detection of lung nodules, but also provides auxiliary reference for clinical diagnosis in the differentiation of benign and malignant lung nodules. However, its accuracy cannot yet replace manual work. Doctors usually use a combination of CT images, laboratory test data, age, gender, smoking history, previous cancer history, family cancer history and other clinical information to assess the malignancy of lung nodules. Therefore, it can be seen that the intelligent graded diagnosis of lung nodules is a multimodal learning problem. Multimodal learning can learn better feature representations by leveraging the complementarity between multimodal data and eliminating redundancy between modalities, thus showing outstanding potential in the differential diagnosis of benign and malignant lung nodules.

[0007] Chinese invention patent application No. 201710230623.1 discloses a deep learning-based lung nodule detection, grading and management method and system. The method includes: acquiring ultra-low-dose spiral CT thin-layer images of the chest, outlining the lung area in the CT image, and marking all lung nodules in the lung area; training a lung area segmentation network, a suspected lung nodule detection network and a lung nodule screening and grading network; obtaining the lung nodule time series of all patients in the image set and their corresponding grading information annotations, and constructing a lung nodule management database; training a lung cancer diagnosis network based on a three-dimensional convolutional neural network and a long short-term memory network. The deep learning-based lung nodule detection, grading and management method and system disclosed in this invention trains a lung region segmentation network, a suspected lung nodule detection network, a lung nodule screening and grading network and a lung cancer diagnosis network, and can accurately detect lung nodules. It has a certain auxiliary diagnostic role in clinical use, but does not comprehensively utilize multimodal data such as CT images, age, gender, smoking history, previous tumor history, family tumor history, etc. to evaluate the malignancy of lung nodules. It only uses CT image information for detection and grading. Its grading accuracy is limited and lacks guiding value for the further clinical treatment of nodules.

[0008] Chinese invention patent application No. 201710340309.9 discloses a method and system for grading lung nodules based on deep learning and data fusion. The method comprises: preprocessing multiple acquired computed tomography (CT) lung images to obtain preprocessed CT lung nodule images; extracting features from each CT lung nodule image and preclassifying each lung nodule image based on the image features; repeatedly training and iterating based on the preclassified lung nodule image features and corresponding clinical data to obtain a diagnostic model; and grading lung nodules based on the diagnostic model. While the lung nodule grading method and system disclosed in this invention utilizes preclassified lung nodule image features and corresponding clinical data, which can improve the accuracy of predicting lung nodule malignancy to a certain extent, since lung nodule image features and non-imaging features from different sources, such as gender, age, and smoking history, contribute differently to the assessment of lung nodule malignancy, how can the image features derived from CT examinations and non-imaging features from clinical data be effectively fused across modalities? This allows for efficient and accurate clinical grading diagnosis of lung nodules according to the Lung-RADS grading standard, achieving a fine-grained four-level classification of lung nodule malignancy, to assist doctors in scientifically evaluating and treating lung nodules, avoiding delayed diagnosis or overdiagnosis of lung cancer, and maximizing patient benefits while achieving rational allocation of medical resources, which remains a challenging problem faced clinically.

[0009] Summary of the Invention

[0010] In order to solve the problems existing in the prior art, the present invention provides a method and system for intelligent grading of lung nodules based on multimodal feature fusion.

[0011] A method for intelligent grading of pulmonary nodules based on multimodal feature fusion, comprising the following steps:

[0012] (1) Design a multi-task feature extraction network based on the attention mechanism to obtain sufficient and discriminative image feature descriptions. Based on chest CT images and corresponding imaging reports, the 2D region of interest (ROI) and 3D volume of interest (VOI) data of lung nodules are obtained. The corresponding radiomics features and deep image features are extracted from the 2D ROI and 3D VOI, respectively. Feature fusion is then performed to obtain sufficient and discriminative image feature descriptions of lung nodules.

[0013] (2) Design a multimodal feature fusion method based on graph representation learning to obtain a total sample feature graph for effective cross-modal feature fusion. Based on CT imaging reports, clinical information, serological examinations, and CT imaging features, a multimodal information extraction method is used to obtain the specific feature representation and graph structure of each modality. Then, the feature representations and graph structures obtained from different modalities are fused using an attention mechanism to obtain a total sample feature graph for cross-modal feature fusion.

[0014] (3) We proposed an optimization and clinical validation method for a self-supervised GCN (Graph Convolutional Network) classification model to form a lung nodule grading diagnostic model. A GCN model for lung nodule grading was constructed, including a classification network, a weight learning network, and a self-supervised network. A total loss function was designed, and the GCN model was trained and optimized to achieve fine-grained grading of lung nodule malignancy with only subtle differences.

[0015] Pulmonary nodule grading was performed according to the pulmonary nodule grading GCN model.

[0016] In one embodiment, the non-image features include feature scales derived from CT examination reports, clinical information, and serum examination data.

[0017] In one embodiment, a multi-task feature extraction network based on an attention mechanism is designed to parallelly compute radiomics features and deep image features based on the 2D ROI and 3D VOI of CT images, and effectively fuse these features using the attention mechanism. During the training phase of the feature fusion layer parameter model, not only the subsequent classification task is considered, but also a CT sign semantic label consistency measurement task is designed for coordinated feedback. By learning and training these two tasks in parallel and sharing the learned feature parameters, sufficient and discriminative image feature descriptions of lung nodules are obtained.

[0018] In one embodiment, the process of acquiring deep image features corresponding to the three-dimensional VOI data includes: inputting the three-dimensional VOI data into a feature extraction backbone network that combines a 3D CNN (3D Convolutional Neural Network) and a Transformer to obtain deep image features that integrate three-dimensional local-global multi-scale structural information.

[0019] In one embodiment, the radiomics features and the deep image features are fused to obtain fused features, including: fusing the radiomics features and the deep image features based on an attention mechanism.

[0020] In one embodiment, the designed multimodal feature fusion method based on graph representation learning includes:

[0021] The multimodal information extraction method is used to obtain the specific feature representation and graph structure of each modality, and then the feature representation and graph structure obtained from different modalities are fused using the attention mechanism to obtain the total sample feature graph data of cross-modal feature fusion.

[0022] Let set X = {X 1 ,X 2 ,…,X M} represents the multimodal feature information of the original input, where m∈[1,M] represents the feature matrix of the mth mode with N samples, and the feature dimension of this mode is d m .

[0023] (1) Multimodal information extraction method

[0024] The purpose of the multimodal information extraction method is to extract specific modal features and graph structures from each modal information. A modal information extraction network is designed here to implement this process.

[0025] For simplicity, we can directly use X=[x1,x2,…,x N ]∈R N×d Represents any selected modal original feature matrix as the input of the modal information extraction network. First, in order to improve the computational efficiency and performance of the model, a learnable function f is used. A The original input feature vector x i Projected into a low-dimensional feature space, f A It is defined as a simple multilayer perceptron (MLP), and the calculation process is shown in formula (1):

[0026] here, represents the learnable weight parameter, and σ represents the nonlinear activation function.

[0027] The correlation between paired samples is calculated by cosine similarity, and the similarity matrix S∈R representing the correlation between the total sample features is obtained N×N , the calculation formula of the similarity between any two sample features is shown in formula (2). For example, the similarity s between sample i and sample j is ij The calculation is as follows:

[0028] The similarity matrix S is subjected to threshold processing as shown in formula (3) to obtain the adjacency matrix of the modality-specific graph structure:

[0029] Here, θ represents a threshold value ranging from 0 to 1.

[0030] The feature matrix X and the graph structure A are sent together into the graph neural network for feature aggregation transformation. The calculation process from each input layer to the output layer of the graph neural network is shown in formula (4):

[0031] Where σ represents the nonlinear activation function, μ1 and μ2 represent the learnable weight coefficients, H represents the two weight matrices to be learned in the l-th layer of the graph neural network, and D represents the degree matrix of the adjacency matrix A. (0) =X, through the feature update of the graph neural network, the final graph neural network output layer H is defined as the optimized feature representation of the modality

[0032] Taking the original feature vector X of each mode as input, the modal information extraction network can be used to obtain the corresponding modal optimized graph structure A and feature representation.

[0033] (2) Cross-modal feature fusion based on attention mechanism

[0034] The cross-modal feature fusion based on the attention mechanism refers to the multimodal feature set obtained from different modalities. And the multimodal graph structure set A={A 1 ,A 2 ,…,A M}, cross-modal feature representation is obtained by fusion through attention mechanism.

[0035] Taking the i-th node as an example, its feature is expressed as Obtain the attention score for each modality based on the attention mechanism:

[0036] First, the multimodal features are transformed by nonlinear transformation, and then a shared attention vector q is used to obtain the attention value

[0037] where W∈R h′×h ,q∈R h′×1 are all learnable weight parameters. After calculating the attention value of each modality, the normalization technique is used to obtain the attention score of all modalities:

[0038] After multiplying the modality’s attention score with the optimized features and graph structure of the corresponding modality, the sum is added to obtain the final fused multimodal features and graph structure:

[0039] (3) Classification prediction based on graph neural network

[0040] The final feature matrix X final And graph structure A final Send it to the graph neural network classifier and output the final classification prediction result.

[0041] In one embodiment, the overall loss function is composed of a weighted cross entropy loss function, a semantic constraint loss function, and a self-supervised contrastive loss function.

[0042] In one embodiment, the overall loss function is as shown in formula (9): L GL =α1L WCE +α2L SCL +α3L SSL (9)

[0043] Where α1, α2, α3 represent the trade-off hyperparameters, L WCE represents the weighted cross entropy loss function, L SCL represents the semantic constraint loss function, L SSL represents the self-supervised contrastive loss function;

[0044] The weighted cross entropy loss function is shown in formula (10):

[0045] The semantic constraint loss function is shown in formula (11):

[0046] The self-supervised contrast loss function is shown in formula (12):

[0047] Here, E represents the number of views, Represents the distance metric between the class label distribution of the i-th view and the class label distribution of the original image.

[0048] A pulmonary nodule intelligent grading system based on multimodal feature fusion includes a processor and a memory, wherein the processor is used to process instructions stored in the memory to implement the above-mentioned pulmonary nodule intelligent grading method based on multimodal feature fusion.

[0049] The present invention has the following beneficial effects: in terms of medical image analysis, a multi-task learning model is designed for the simultaneous extraction and fusion of three-dimensional CNN convolutional features and semantic features such as two-dimensional textures, and a multi-dimensional and multi-category image feature effective fusion module is designed based on the attention mechanism to obtain semantically consistent fusion of local-global multi-scale structural information and accurate feature description, which provides a new method for small target feature extraction based on three-dimensional CT images; in terms of multimodal data analysis, by proposing a multimodal optimization feature map construction based on adaptive graph representation learning and an effective fusion method of cross-modal feature maps based on the attention mechanism, a class of high-precision cross-modal feature fusion GCN multi-classification algorithms is designed to achieve fine grading of the malignancy of lung nodules with only slight differences, providing a new approach to the design of fine-grained classification algorithms; in terms of solving clinical problems, a classification algorithm optimization method and clinical verification scheme that organically combines generative and contrastive self-supervised learning are proposed, which can form an intelligent diagnosis model for lung nodules, which is expected to assist doctors in scientifically evaluating and treating lung nodules, avoid delayed diagnosis or overdiagnosis of lung cancer, and promote the standardization of clinical diagnosis and treatment of lung nodules. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] FIG1 is an overall flow chart of an intelligent grading method for pulmonary nodules based on multimodal feature fusion provided by the present invention;

[0051] FIG2 is a schematic diagram of the overall architecture corresponding to a method for intelligent grading of pulmonary nodules based on multimodal feature fusion provided by the present invention;

[0052] Figure 3 is a schematic diagram of the network corresponding to accurate feature extraction;

[0053] Figure 4 is a network diagram corresponding to the classification algorithm;

[0054] FIG5 is a network diagram corresponding to algorithm verification and optimization. DETAILED DESCRIPTION

[0055] This embodiment provides an intelligent grading method for lung nodules based on multimodal feature fusion, as shown in Figure 1. The overall concept of the intelligent grading method for lung nodules based on multimodal feature fusion provided in this embodiment is as follows: design a multi-task learning model for accurate extraction and effective fusion of multi-dimensional and multi-class CT image features based on the attention mechanism; construct cross-modal feature effective fusion graph data of multi-source heterogeneous data, and propose a graph convolutional neural network lung nodule grading algorithm based on adaptive graph representation learning; and design a lung nodule grading algorithm optimization and clinical verification scheme based on self-supervised learning.

[0056] This embodiment designs the implementation route of this embodiment according to the main line of "precise feature extraction - classification algorithm design - algorithm verification and optimization", and its overall architecture is shown in Figure 2. As can be seen from the technical route shown in Figure 2, first, the two-dimensional region of interest (ROI) and three-dimensional region of interest (VOI) data of the lung nodules are obtained based on the chest CT examination image and the corresponding examination report, and the required two-dimensional ROI and three-dimensional VOI input data of the lung nodules are obtained; secondly, the feature precision extraction model is used to obtain the required CT image features; then, the image features derived from the CT image, CT image report, clinical information and non-image features of serological examination are used as multi-source heterogeneous and multi-modal input data to design a multi-classification algorithm; then, the classification algorithm is further improved using the algorithm verification and optimization scheme to form the final fine grading model for the malignancy of lung nodules.

[0057] The following are the steps for implementing a method for intelligently grading pulmonary nodules based on multimodal feature fusion, provided in this embodiment:

[0058] Step 1: Obtain 2D ROI data and 3D VOI data of the lung nodule, and obtain the radiomics features corresponding to the 2D ROI data and the deep image features corresponding to the 3D VOI data:

[0059] Acquire 2D ROI and 3D VOI data of lung nodules. Based on a multi-task learning framework, calculate 2D CT semantic features based on radiomics and 3D VOI features that fuse local-global multiscale structural information by combining CNN and Transformer. Utilize the Transformer model to fully extract spatially correlated 3D CNN multiscale features, construct a shared feature subspace method that preserves semantic features, and design a multi-task learning model based on the simultaneous extraction and fusion of 3D CNN convolutional features and semantic features such as 2D textures, achieving effective fusion and complementarity of CT-based lung nodule information.

[0060] Figure 3 shows the technical roadmap for precise feature extraction. Based on the acquired 2D ROI and 3D VOI data of lung nodules, radiomics and other methods will be used to calculate CT features such as nodule location, size, calcification, and texture. A feature extraction backbone network combining 3D CNN and Transformer will be designed to obtain deep image features that integrate 3D local-global multi-scale structural information. The 3D VOI data will be input into the 3D CNN and Transformer feature extraction backbone network to obtain deep image features that integrate 3D local-global multi-scale structural information.

[0061] Step 2: Fuse the radiomics features and deep image features to obtain fused features:

[0062] The imaging genomics features and deep image features are fused based on the attention mechanism. At present, the commonly used feature layer fusion is the direct splicing of different types of features, which can easily lead to the problem of "semantic gap" between features. In particular, most of the existing research methods simply train the feature fusion module based on the subsequent classification task, without considering the possible semantic inconsistency between various features, so that the problem of "mutual exclusion" of features may occur in the process of fusion of multiple features. Therefore, this embodiment performs multi-view information fusion based on the Cross View Attention Block structure, fully considering the correlation between multi-dimensional and multi-category features, and designs an effective fusion module of multiple types of features based on CT images on the basis of introducing the consistency measurement of CT sign semantic labels.

[0063] During the model training stage of learning the feature fusion layer parameters, not only the subsequent classification task is considered, but also a CT sign semantic label consistency measurement task is designed for coordinated feedback. The two tasks are learned and trained in parallel and share the learned feature parameters, so that an image feature description of lung nodules that is sufficient, discriminative, and robust can be obtained.

[0064] Step 3: Based on CT imaging reports, clinical information, serological tests, and CT imaging features, a multimodal information extraction method is designed to obtain the feature representation and graph structure of each modality. Then, the feature representation and graph structure obtained from different modalities are fused using the attention mechanism to obtain the total sample feature graph of cross-modal feature fusion.

[0065] Let set X = {X 1 ,X 2 ,…,X M} represents the multimodal feature information of the original input, where m∈[1,M] represents the feature matrix of the mth mode with N samples, and the feature dimension of this mode is d m The multimodal information extraction method aims to extract specific modal features and graph structures from each modal information. A modal information extraction network is designed here to implement this process.

[0066] For simplicity, we can directly use X=[x1,x2,…,x N ]∈R N×d Represents the original feature matrix of any selected mode, which serves as the input of the modal information extraction network.

[0067] In order to improve the computational efficiency and performance of the model, we first use a learnable function f A The original input feature vector x i Projected into a low-dimensional feature space, f AIt is defined as a simple multi-layer perceptron (MLP), and the calculation process is as follows:

[0068] in represents the learnable weight parameter, and σ represents the nonlinear activation function.

[0069] The correlation between paired samples is calculated by cosine similarity, and the similarity matrix S∈R representing the correlation between the total sample features is obtained N×N , the calculation formula of the similarity between any two sample features is as follows. For example, the similarity s between sample i and sample j is ij The calculation is as follows:

[0070] The similarity matrix S is subjected to threshold processing as shown in the following formula to obtain the adjacency matrix of the modality-specific graph structure:

[0071] Where θ represents a threshold value ranging from 0 to 1.

[0072] The feature matrix X and the graph structure A are sent together into the graph neural network for feature aggregation transformation. The calculation process from each input layer to the output layer of the graph neural network is shown in the following formula:

[0073] Where σ represents the nonlinear activation function, μ1 and μ2 represent the learnable weight coefficients, H represents the two weight matrices to be learned in the l-th layer of the graph neural network, and D represents the degree matrix of the adjacency matrix A. (0) =X, through the feature update of the graph neural network, the final graph neural network output layer H is defined as the optimized feature representation of the modality

[0074] Taking the original eigenvector X of each mode as input, the designed modal information extraction network is used to obtain the corresponding modal optimized graph structure A and feature representation.

[0075] The cross-modal feature fusion based on the attention mechanism refers to the multimodal feature set obtained from different modalities. And the multimodal graph structure set A={A 1 ,A 2 ,…,A M}, feature representation is obtained by fusion through attention mechanism.

[0076] Taking the i-th node as an example, its feature is expressed as Obtain attention scores for each modality based on the attention mechanism

[0077] First, the multimodal features are transformed by nonlinear transformation, and then a shared attention vector q is used to obtain the attention value

[0078] where W∈R h′×h ,q∈R h′×1 are all learnable weight parameters. After calculating the attention value of each modality, the normalization method is used to obtain the attention score of all modalities:

[0079] After multiplying the modality’s attention score with the optimized features and graph structure of the corresponding modality, the sum is added to obtain the final fused multimodal features and graph structure:

[0080] The final feature matrix X final And graph structure A final Send it to the graph neural network classifier and output the final classification prediction result.

[0081] In this embodiment, the non-image features include CT examination reports, clinical information, and serum examination data.

[0082] To address the problem that multimodal features of data from different sources have different impacts on classification performance, this embodiment uses nodule image features and non-image features from three different sources to construct multiple feature maps, and proposes a multi-map information fusion method based on the attention mechanism.

[0083] This embodiment proposes a joint training strategy with the GCN classification model. While constructing a cross-modal feature graph on the front end through graph representation learning, it can be jointly optimized with the GCN classification model to reveal the intrinsic connections between different subjects and adaptively extract more information representing the associations between different subjects, effectively improving classification performance. To address the problem of different feature propagation speeds at different nodes in the GCN classification algorithm during training, which can easily lead to asynchronous feature updates at edge nodes in the network, a method is proposed that combines jump connections and a multi-head attention mechanism to adaptively select an appropriate propagation range for each node.

[0084] For classification algorithms based on multimodal learning, the problem that different modal features have different effects on the classification performance of the entire model is often ignored. This embodiment provides a classification algorithm design technology route as shown in Figure 4.

[0085] Graph structure learning is performed for different modal features respectively. Through multiple graph representation learning and graph convolution layer channels, information aggregation is first performed separately within each modality. Then, the attention mechanism is used to realize information interaction and aggregation between modalities, thereby extracting effective image features of nodes and sending them to the downstream classification model to complete accurate grading prediction of lung nodules.

[0086] In the process of graph structure modeling, adaptive graph representation learning and training will be performed to optimize the structure of the input graph by calculating metric learning methods such as Gaussian distance or cosine distance between features, and information aggregation strategies will be used to further improve the accuracy of the classification algorithm.

[0087] Step 4: Construct a GCN model for pulmonary nodule grading. The GCN model includes a classification network, a weight learning network, and a self-supervisory network. Based on the classification network, the weight learning network, and the self-supervisory network, the overall loss function is obtained:

[0088] Graph data augmentation can create new graph data through certain transformations without changing the semantics, and use the created graph data to collaboratively train GCN, thereby improving classification performance and robustness. Common methods include node dropping, edge perturbation, changing node attribute masks, and random walk sampling subgraphs. Since doctors often save some typical case data and improve clinical diagnostic accuracy through comparative diagnosis, this embodiment proposes a method for creating collaborative training graph data based on typical case data while ensuring the semantic consistency of the graph, and designs a semantic constraint loss function design method for measuring the degree of closeness between the class distribution under different views and the original graph semantics.

[0089] Loss function design for model optimization: The prerequisite for graph data enhancement is to ensure the consistency of graph semantics, that is, the predicted labels cannot change significantly due to data enhancement. In self-supervised contrastive learning, it is required to maximize the similarity of positive pairs and minimize the similarity of negative pairs to increase the class distance. Therefore, this embodiment first designs a weighted cross entropy loss function, a semantic constraint loss, and a self-supervised contrast loss function to obtain an improved total loss; then, the trained model is integrated into the independently developed "Medical Image Storage and Transmission System Software" product, and deployed to the hospital for on-site trial evaluation, so as to further better utilize the expert prior knowledge of clinicians and improve the overall loss function. Tune the model and improve its generalization performance.

[0090] In real-world scenarios, the number of samples of different levels of lung nodules is seriously unbalanced. To address this issue and further improve the performance and versatility of the algorithm, this embodiment provides an algorithm optimization technology route based on self-supervised learning, as shown in Figure 5. It includes three components: (a) classification network;

[0091] (b) Weight learning network; (c) Self-supervised network.

[0092] The overall loss function of the entire classification network is composed of the weighted cross entropy loss function, the semantic constraint loss function and the self-supervised contrast loss function, as shown below: GL =α1L WCE +α2L SCL +α3L SSL ;

[0093] Where α1, α2, α3 represent the trade-off hyperparameters, L WCE represents the weighted cross entropy loss function, L SCL represents the semantic constraint loss function, L SSL represents the self-supervised contrastive loss function;

[0094] By designing a weight network to learn the weight of each sample, and using the weight network and the classification network to compete with each other to optimize the weighted cross entropy loss function shown below, the class imbalance problem can be alleviated to a certain extent.

[0095] In designing the self-supervised network, we aim to further expand supervisory information through graph data augmentation while ensuring semantic consistency. Furthermore, we aim to increase the inter-class distance of the classification algorithm through self-supervised contrastive learning. This further improves algorithm performance and robustness. The semantic constraint loss function and self-supervised contrastive loss function are shown below:

[0096] Here, E represents the number of views, Represents the distance metric between the class label distribution of the i-th view and the class label distribution of the original image.

[0097] Step 5: Perform pulmonary nodule grading according to the pulmonary nodule grading GCN model:

[0098] Lung nodule grading detection is performed based on the lung nodule grading GCN model trained in step 4.

[0099] This embodiment also provides an intelligent grading system for pulmonary nodules based on multimodal feature fusion, including a processor and a memory. The processor is configured to process instructions stored in the memory to implement an intelligent grading method for pulmonary nodules based on multimodal feature fusion. Since an intelligent grading method for pulmonary nodules based on multimodal feature fusion has been described in detail in the above embodiment, it will not be repeated here.

[0100] Therefore, to address the clinical challenge of "difficulty in accurately grading the malignancy of lung nodules," we leveraged real-world clinical multimodal data and addressed the key scientific challenge of "accurately extracting and effectively integrating features from multimodal data." We designed an efficient, highly accurate, and continuously learning multi-classification algorithm for lung nodules. Through algorithm optimization and clinical validation, we developed an intelligent lung nodule diagnosis model. This model will assist physicians in correctly managing the growing population of lung nodules, promote the rational allocation of medical resources, and standardize the clinical diagnosis and treatment of lung nodules.

[0101] The specific beneficial effects are as follows:

[0102] (1) In the field of medical image analysis, a multi-task learning model is designed to simultaneously extract and fuse 3D CNN convolutional features and semantic features such as 2D texture. A multi-dimensional and multi-class image feature fusion module is designed based on the attention mechanism to obtain semantically consistent and accurate feature descriptions of the fused local-global multi-scale structural information. This provides a new method for small target feature extraction based on 3D CT images.

[0103] (2) In terms of multimodal data analysis, by proposing a multi-graph construction method based on adaptive graph representation learning and an effective multi-graph information fusion method based on attention mechanism, a high-precision cross-modal feature fusion GCN multi-classification algorithm is designed to achieve fine grading of the malignancy of lung nodules with only slight differences, providing a new approach for the design of fine-grained classification algorithms.

[0104] (3) In terms of solving clinical problems, a classification algorithm optimization method and clinical verification scheme that organically combines generative and contrastive self-supervised learning are proposed, which can form an intelligent diagnosis model for lung nodules, and is expected to assist doctors in scientifically evaluating and treating lung nodules, avoid delayed diagnosis or overdiagnosis of lung cancer, and promote the standardization of clinical diagnosis and treatment of lung nodules.

Claims

1. A method for intelligent grading of pulmonary nodules based on multimodal feature fusion, characterized in that: The steps include: (1) Design a multi-task feature extraction network based on the attention mechanism to obtain sufficient and discriminative image feature descriptions; obtain the 2D region of interest (ROI) and 3D region of interest (VOI) data of lung nodules based on chest CT images and corresponding imaging reports, extract the corresponding imaging features and deep image features from the ROI and VOI, and perform feature fusion to obtain sufficient and discriminative image feature descriptions of lung nodules; (2) Design a multimodal feature fusion method based on graph representation learning to obtain a total sample feature graph of cross-modal feature fusion. Based on CT image reports, clinical information, serological examinations, and CT image features, a multimodal information extraction method is used to obtain the specific feature representation and graph structure of each modality. Then, the feature representation and graph representation structure obtained from different modalities are fused using an attention mechanism to obtain a total sample feature graph of cross-modal feature fusion. (3) Propose an optimization and clinical validation method based on the self-supervised learning GCN classification model to form a lung nodule grading diagnosis model; construct a lung nodule grading GCN model, including a classification network, a weight learning network, and a self-supervised network, design a total loss function, train and optimize the lung nodule grading GCN model, and achieve fine grading of lung nodule malignancy with only slight differences; Pulmonary nodule grading was performed according to the pulmonary nodule grading GCN model.

2. The method for intelligent grading of pulmonary nodules based on multimodal feature fusion according to claim 1, characterized in that: A method for obtaining a total sample feature map for input into a graph convolutional neural network classifier GCN, wherein the non-image features include feature scales derived from CT examination reports, clinical information, and serum examination data.

3. The method for intelligent grading of pulmonary nodules based on multimodal feature fusion according to claim 1, characterized in that: The proposed method for accurate extraction of multiple types of image features based on CT examination data parallelly calculates radiomics features and deep image features based on two-dimensional ROI and three-dimensional VOI, respectively, and effectively fuses radiomics features and deep image features based on the attention mechanism; in the training stage of learning the feature fusion layer parameter model, not only the subsequent classification task is considered, but also the CT sign semantic label consistency measurement task is designed for coordinated feedback. By learning and training the two tasks in parallel and sharing the learned feature parameters, a sufficient and discriminative image feature description of lung nodules is obtained.

4. The method for intelligent grading of pulmonary nodules based on multimodal feature fusion according to claim 1, characterized in that: The process of acquiring the deep image features corresponding to the three-dimensional VOI data includes: inputting the three-dimensional VOI data into a feature extraction backbone network combining 3D CNN and Transformer to acquire the deep image features that integrate three-dimensional local-global multi-scale structural information.

5. The method for intelligent grading of pulmonary nodules based on multimodal feature fusion according to claim 1, characterized in that: The designed multimodal feature fusion method based on graph representation learning first uses the multimodal information extraction method to obtain the specific feature representation and graph structure of each modality, and then uses the attention mechanism to fuse the feature representation and graph structure obtained from different modalities to obtain the total sample feature graph data of cross-modal feature fusion; Let the set X = {X 1 ,X 2 ,…,X M } represents the multimodal feature information of the original input, where m∈[1,M] represents the feature matrix of the mth mode with N samples, and the feature dimension of this mode is d m ; (1) Multimodal Information Extraction Method The multimodal information extraction method aims to extract specific modal features and graph structures from each modal information. Here, a modal information extraction network is designed to implement the process. For simplicity, we can directly use X = [x1, x2, …, x N ]∈R N×d Represents the original feature matrix of any selected modal as the input of the modal information extraction network; in order to improve the computational efficiency and performance of the model, first a learnable function f is used A The original input feature vector x i Projected into a low-dimensional feature space, f A It is defined as a simple multi-layer perceptron (MLP), and the calculation process is shown in formula (1): in represents the learnable weight parameter, σ represents the nonlinear activation function; Then, the correlation between paired samples is calculated by cosine similarity to obtain the similarity matrix S∈R that represents the correlation between the total sample features. N×N , the similarity calculation formula between any two sample features is shown in formula (2); for example, the similarity s between sample i and sample j ij The similarity matrix S is subjected to threshold processing as shown in formula (3) to obtain the adjacency matrix of the modality-specific graph structure: Where θ represents a threshold value ranging from 0 to 1; The feature matrix X and the graph structure A are sent to the graph neural network for feature aggregation transformation. The calculation process from each input layer to the output layer of the graph neural network is shown in formula (4): Where σ represents the nonlinear activation function, μ1 and μ2 represent the learnable weight coefficients, represents the two weight matrices to be learned in the l-th layer of the graph neural network, D represents the degree matrix of the adjacency matrix A; H (0) =X, through the feature update of the graph neural network, the final graph neural network output layer H is defined as the optimized feature representation of the modality For each mode's original feature vector X as input, the above-designed modal information extraction network is used to obtain the corresponding modal optimized graph structure A and feature representation (2) Cross-modal feature fusion based on attention mechanism The cross-modal feature fusion based on the attention mechanism refers to the multimodal feature set obtained from different modalities. and the multimodal graph structure set A = {A 1 ,A 2 ,…,A M }, obtain cross-modal feature representation through fusion through attention mechanism; Taking the i-th node as an example, its feature is expressed as Obtain the attention score for each modality based on the attention mechanism: The multimodal features are transformed through nonlinear transformation, and then a shared attention vector q is used to obtain the attention value where W∈R h′×h , q∈R h′×1 are all learnable weight parameters; after calculating the attention value of each modality, normalization is used to obtain the attention scores of all modalities: After multiplying the attention score of the modality with the optimized features and graph structure of the corresponding modality, the sum is added to obtain the final fused multimodal features and graph structure: (3) Classification prediction based on graph neural network The final feature matrix X final And the graph structure A final Send it to the graph neural network classifier and output the final classification prediction result.

6. The method for intelligent grading of pulmonary nodules based on multimodal feature fusion according to claim 1, characterized in that: The multimodal information extraction method obtains an optimized feature representation and graph structure for each modality, where the number of modalities and feature dimensions are set arbitrarily.

7. The method for intelligent grading of pulmonary nodules based on multimodal feature fusion according to claim 1, characterized in that: The cross-modal feature fusion method based on the attention mechanism learns according to the end-to-end graph structure representation and obtains the contribution score of each modal feature.

8. The method for intelligent grading of pulmonary nodules based on multimodal feature fusion according to claim 1, characterized in that: An optimization and clinical verification method based on self-supervised learning GCN classification model is proposed, and a lung nodule grading diagnosis model is formed; a lung nodule grading GCN model is constructed, including a classification network, a weight learning network and a self-supervised network.

9. The method for intelligent grading of pulmonary nodules based on multimodal feature fusion according to claim 1, characterized in that: The total loss function is composed of a weighted cross entropy loss function, a semantic constraint loss function, and a self-supervised contrast loss function; The overall loss function is shown in formula (9): L GL =α1L WCE +α2L SCL +α3L SSL (9) Among them, α1, α2, α3 represent the trade-off hyperparameters, L WCE represents the weighted cross entropy loss function, L SCL represents the semantic constraint loss function, L SSL represents the self-supervised contrastive loss function; The weighted cross entropy loss function is as follows: The semantic constraint loss function is as follows: The self-supervised contrast loss function is as follows: Where E represents the number of views, Represents the distance metric between the i-th view class label distribution and the original image class label distribution.

10. An intelligent grading system for pulmonary nodules based on multimodal feature fusion, characterized in that: It comprises a processor and a memory, wherein the processor is used to process instructions stored in the memory to implement the intelligent grading method for pulmonary nodules based on multimodal feature fusion as described in any one of claims 1-9.

Images