Lung cancer image and text auxiliary analysis system and method based on multi-modal fusion

Abstract

The invention discloses a lung cancer image and text auxiliary analysis system and method based on multi-modal fusion. The system comprises a multi-source data preprocessing module, a single-modal deep learning feature extraction module, a multi-modal fusion module and a diagnosis and interpretability output module. The method comprises the following steps: S1, carrying out preprocessing operation on a lung cancer CT image of a patient through a multi-source data preprocessing module; s2, performing preprocessing operation on the clinical text of the patient through a multi-source data preprocessing module; s3, performing feature extraction operation on the preprocessed lung cancer CT image through a single-mode deep learning feature extraction module to obtain image features; s4, performing feature extraction operation on the preprocessed clinical text through a single-mode deep learning feature extraction module to obtain text features; s5, performing fusion operation on the image features and the text features to obtain fusion features; and S6, a diagnosis and interpretability output module carries out analysis and outputs a diagnosis result.

Description

Technical Field

[0001] This invention relates to the field of medical imaging, and more particularly to a lung cancer image and text-assisted analysis system and method based on multimodal fusion. Background Technology

[0002] The current field of lung cancer imaging and clinical data processing suffers from the following core technological deficiencies, making it difficult to meet the clinical needs for auxiliary screening and subtype assessment of lung cancer:

[0003] Limitations of single-modal data processing: Existing technologies mostly process CT images or clinical text data separately. CT image processing only focuses on the morphological characteristics of lesions and lacks related information such as patient history and laboratory tests. It is not capable of identifying the characteristics of early occult lesions such as small nodules (≤3mm) and ground-glass nodules (GGN). Clinical text processing can only extract semantic information and cannot establish a correlation with the lesion characteristics in the images. When used alone, it has obvious analytical blind spots.

[0004] Insufficient multimodal fusion technology: A few studies involving multimodal data only use basic fusion strategies such as "early feature stitching" and "simple weighted average", without building a deep semantic interaction mechanism between images and text. This makes it impossible to effectively mine cross-modal correlation information (such as the potential link between nodular spiculation in CT images and the patient's smoking history), resulting in poor robustness of the model in analyzing complex cases (such as patients with multiple underlying diseases).

[0005] Poor model generalization and interpretability: Traditional machine learning methods require tedious manual feature engineering, have poor adaptability to multi-center, large-scale data, and weak generalization ability of analysis results; although some deep learning models have improved feature recognition accuracy, they lack visualization and interpretation modules, making it difficult for clinicians to trace the basis of analysis, resulting in low trust in model results and hindering the transformation of technology into clinical support scenarios.

[0006] Furthermore, early symptoms of lung cancer are often subtle, and traditional methods of relying on manual image interpretation and medical history review are inefficient and have a high rate of missed diagnoses. As a result, 70% of patients are diagnosed at an advanced stage, with a 5-year survival rate of only 19.7%. There is an urgent need to improve the efficiency and accuracy of lung cancer-related data analysis through technological innovation to provide auxiliary support for early clinical intervention. Summary of the Invention

[0007] The purpose of this invention is to address the above-mentioned problems by providing a lung cancer image and text auxiliary analysis system and method based on multimodal fusion, which can realize collaborative processing and auxiliary analysis of CT images and clinical text data and effectively reduce the rate of missed diagnosis and treatment.

[0008] To achieve the above objectives, the technical solution of the present invention is as follows:

[0009] The lung cancer image and text-assisted analysis system based on multimodal fusion includes a multi-source data preprocessing module for preprocessing lung cancer CT images and clinical texts, a single-modal deep learning feature extraction module for extracting features from lung cancer CT images and clinical texts, a multimodal fusion module for fusing the features extracted from lung cancer CT images and clinical texts, and a diagnostic and interpretability output module for outputting analysis results and assisting in diagnosis. The signal output terminal of the multi-source data preprocessing module is connected to the signal input terminal of the single-modal deep learning feature extraction module, the signal output terminal of the single-modal deep learning feature extraction module is connected to the signal input terminal of the multimodal fusion module, and the signal output terminal of the multimodal fusion module is connected to the signal input terminal of the diagnostic and interpretability output module.

[0010] A method for lung cancer image and text-assisted analysis based on multimodal fusion includes the following steps:

[0011] S1. Perform preprocessing operations on the patient's lung cancer CT images through the multi-source data preprocessing module;

[0012] S2. Preprocess the patient's clinical text using the multi-source data preprocessing module;

[0013] S3. The preprocessed lung cancer CT images are subjected to feature extraction operations through a single-modal deep learning feature extraction module to obtain image features;

[0014] S4. The preprocessed clinical text is subjected to feature extraction operation through the single-modal deep learning feature extraction module to obtain text features;

[0015] S5. The image features and text features are fused using the multimodal fusion module to obtain fused features;

[0016] S6. Input the fused features into the diagnostic and interpretability output module, which then analyzes and outputs the diagnostic results.

[0017] Furthermore, step S1 specifically includes the following steps:

[0018] S11. Standardize the grayscale range of lung cancer CT images. The calculation formula is as follows:

[0019] ;

[0020] in, The original lung cancer CT image shows voxel gray values. For lung window position, For the width of the lung window, This means restricting the result within the parentheses to the interval [0,1]. The standardized grayscale value;

[0021] S12. An improved U-Net architecture is used for lung field segmentation, and the Dice loss function is used to optimize the segmentation accuracy. The calculation formula is as follows:

[0022] ;

[0023] in, To predict lung field regions for the model, To establish the gold standard for physicians;

[0024] S13. Resample the original lung cancer CT images using linear interpolation. Voxels; samples are expanded using elastic deformation, and the formula for calculating elastic deformation is:

[0025] ;

[0026] in, It is a Gaussian random displacement field.

[0027] Furthermore, step S2 specifically includes the following steps:

[0028] S21. Entities are extracted from clinical text based on BERT-like models. The output layer incorporates a conditional random field (CRF) to optimize entity boundaries. The loss function of the CRF is:

[0029] ;

[0030] in, For state transition weights, The entity label score output by BioBERT;

[0031] S22. The extracted entities are mapped to SNOMED CT term codes to achieve term standardization; then, the standardized term codes are transformed into a high-dimensional semantic feature vector sequence through a word embedding layer.

[0032] Furthermore, step S3 specifically includes the following steps:

[0033] S31. Construct an image feature extraction model to perform feature extraction operations on the preprocessed lung cancer CT images to obtain image features; the image feature extraction model consists of residual blocks, window attention modules, channel attention modules, and spatial attention modules.

[0034] S32. Residual blocks alleviate gradient vanishing through cross-layer connections; the calculation formula is as follows:

[0035] ;

[0036] in, Dimension matching implemented for 1×1×1 convolutions;

[0037] S33. A feature extraction network based on the Transformer architecture extracts global and local features in stages. The calculation formula is as follows:

[0038] ;

[0039] in, The feature matrix within the window. This is the relative position offset matrix;

[0040] S34. The channel attention module adaptively adjusts the channel weights, and its calculation formula is as follows:

[0041] ;

[0042] in, This is the result of global average pooling. For the weights of the fully connected layer, For the Sigmoid function;

[0043] S35. The spatial attention module generates a spatial weight map, and its calculation formula is as follows:

[0044] ;

[0045] in, The spatial weight map is the output of a 7×7 convolution.

[0046] Furthermore, step S4 specifically includes the following steps:

[0047] S41. A Bidirectional Long Short-Term Memory (BiLSTM) network is used to process the high-dimensional semantic feature vector sequence to capture long-range contextual dependencies between clinical terms and to memorize sequence information through hidden concatenation; the calculation formula is as follows:

[0048] ;

[0049] in, It is in a forward and backward hidden state;

[0050] S42. Use the window attention module to enhance the association of clinical terms;

[0051] S43. Focusing on high-risk information, the field weights are calculated using the following formula:

[0052] ;

[0053] in, For text field features, For learnable weights, Assign field weights;

[0054] S44. Generate text features by weighting the 512-dimensional text feature vector according to the field weights.

[0055] Furthermore, step S5 specifically includes the following steps:

[0056] S51. Taking image features as the primary factor and explicitly associating them with text features, the calculation formula is as follows:

[0057] ;

[0058] in, Image features For text features, Modal adaptation weight matrix;

[0059] S52. By using tensor low-rank decomposition, the parameter scale is reduced while preserving the high-order interaction between image features and text features. The calculation formula is as follows:

[0060] ;

[0061] in, For low-dimensional core tensors, For image feature projection matrix, The text feature projection matrix, To output the projection matrix, This is a tensor outer product operation;

[0062] S53. Simultaneously optimize benign / malignant classification and subtype identification; the calculation formula is as follows:

[0063] ;

[0064] S54. Calculate the focus loss. The formula is as follows:

[0065] ;

[0066] in, Predict probabilities for the model;

[0067] S55. Contrastive learning regularization term, which enhances feature discriminative power, is calculated using the following formula:

[0068] ;

[0069] in, For sample features, Features for positive / negative examples This represents the set of negative samples in the current batch. For cosine similarity, This is the temperature coefficient.

[0070] Furthermore, in step S6, the diagnosis and interpretability output module outputs the probability of benign or malignant transformation and the pathological subtype, and visualizes the key areas of lung cancer CT images through Grad-CAM. At the same time, it sorts the keywords in the clinical text according to attention weights to improve clinical credibility.

[0071] Compared with the prior art, the advantages and positive effects of this invention are:

[0072] 1. The multimodal fusion module of the present invention significantly improves the diagnostic performance in early lung cancer screening tasks, greatly optimizes the ability to distinguish between benign and malignant nodules and the accuracy of pathological subtype identification, and significantly enhances the ability to detect early small lesions, effectively making up for the diagnostic blind spots of single-modal data.

[0073] 2. This invention significantly shortens the diagnostic time, and its diagnostic efficiency is significantly improved compared to manual image reading; it also reduces the false positive rate and unnecessary follow-up examinations; at the same time, it can help detect early lung cancer, giving patients the best treatment opportunity and playing an important role in improving patient survival rates.

[0074] 3. The modules in this invention support lightweight deployment, can be adapted to the equipment conditions of primary healthcare institutions, and alleviate the problem of uneven distribution of medical resources. Attached Figure Description

[0075] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0076] Figure 1 This is a system overall framework diagram of the present invention;

[0077] Figure 2 This is a schematic diagram of the modal network structure of CT images;

[0078] Figure 3 This is a schematic diagram of the modal network structure of clinical text.

[0079] Figure 4 Here is a flowchart of the Cross-Modal Attention process;

[0080] Figure 5 This is a diagram illustrating the breakdown of Tucker. Detailed Implementation

[0081] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, any modifications, equivalent substitutions, improvements, etc., made by those skilled in the art to all other embodiments obtained without creative effort should be included within the protection scope of the present invention.

[0082] like Figure 1 As shown, this invention discloses a lung cancer image and text-assisted analysis system and method based on multimodal fusion. It achieves collaborative processing and assisted analysis of CT images and clinical text data through a hierarchical architecture of "multi-source data standardization - single-modal feature extraction - multimodal deep fusion - auxiliary analysis output." The specific technical solution is as follows:

[0083] I. System Overall Architecture

[0084] The lung cancer image and text-assisted analysis system based on multimodal fusion comprises four core modules: a multi-source data preprocessing module, a single-modal deep learning feature extraction module, a multimodal fusion module, and a diagnosis and interpretability output module. These modules are linked through standardized data interfaces to form an end-to-end intelligent analysis process for lung cancer-related data.

[0085] The functions and working processes of each module are described in detail below:

[0086] 1. Multi-source data preprocessing module

[0087] (1) CT image preprocessing, such as Figure 2 As shown;

[0088] Window width / window level standardization: unifying the grayscale value range for scanning from different devices, using the following formula:

[0089]

[0090] in, This represents the original CT voxel grayscale values. (Lung window position) (Lung window width) This means that the result within the parentheses is restricted to the interval [0,1] (0 for values ​​less than 0 and 1 for values ​​greater than 1). The standardized grayscale values ​​(0-255) eliminate grayscale shifts caused by device differences.

[0091] Lung field segmentation (FusionLungNet): Employs an improved U-Net architecture, using the Dice loss function to optimize segmentation accuracy, as shown in the following formula:

[0092] ;

[0093] in, To predict lung field regions for the model, To provide physicians with the gold standard, ensure that the segmentation Dice coefficient is ≥0.95 and remove interference from bones, metal artifacts, etc.

[0094] Thin-slice reconstruction and data enhancement: resampling the original CT scan using linear interpolation. Voxels; random rotation (±15°), elastic deformation (deformation coefficient) Expanding the sample, the elastic deformation formula is:

[0095] ;

[0096] in, A Gaussian random displacement field is used to simulate the image characteristics of different scanning positions.

[0097] (2) Clinical text preprocessing, such as Figure 3 As shown;

[0098] Entity Recognition (BioBERT+CRF): Based on BERT-like models, this method extracts entities such as smoking history and tumor markers. The output layer incorporates a Conditional Random Field (CRF) to optimize entity boundaries. The CRF loss function is:

[0099] ;

[0100] in, For state transition weights, The entity label score output by BioBERT is used to ensure that the entity recognition F1-score is ≥ 0.9.

[0101] Structured encoding: The identified entities are mapped to SNOMED CT term codes to achieve term standardization; then, the standardized term codes are transformed into a high-dimensional semantic feature vector sequence through a word embedding layer.

[0102] 2. Single-modal deep learning feature extraction module

[0103] (1) Image feature extraction model, such as Figure 2 As shown;

[0104] ResNet residual blocks: mitigate gradient vanishing through cross-layer connections; residual formula:

[0105] ;

[0106] in, It is usually composed of three stacked layers (1×1×1 dimensionality reduction, 3×3×3 feature extraction, 1×1×1 dimensionality increase). Dimension matching is achieved through 1×1×1 convolution to capture spatial continuity features of lesions (such as nodule size and density).

[0107] The feature extraction network based on the Transformer architecture extracts global and local features in stages. The window attention formula is as follows:

[0108] ;

[0109] in, The feature matrix within the window ( (for feature dimensions) It is a relative position bias matrix used to enhance position awareness. An additional attention mask is added to the shift window to avoid attention calculations across windows. Hierarchical feature fusion is achieved through patch merging to enhance the identification of low-contrast lesions such as GGN.

[0110] Attention Enhancement (SE+CBAM):

[0111] SE module (Channel Attention): Adaptively adjusts channel weights, formula:

[0112] ;

[0113] in, This is the result of global average pooling. For the weights of the fully connected layer, This is a Sigmoid function that highlights key channels such as CT values ​​and textures.

[0114] CBAM module (Spatial Attention): Generates a spatial attention map, formula:

[0115] ;

[0116] in, The spatial weight map output by the 7×7 convolution is multiplied element-wise with the input features to suppress background noise and focus on the nodule region.

[0117] (2) Text feature extraction models, such as Figure 3 As shown;

[0118] BiLSTM Sequence Modeling: This method employs a bidirectional long short-term memory (BiLSTM) network to capture long-range dependencies in text, such as causal or parallel relationships in medical descriptions like "cough with blood in sputum." BiLSTM uses a gating mechanism to address the vanishing gradient problem, effectively memorizing sequence information.

[0119] Forward hidden state ( ): Computed by the feedforward LSTM unit, from the past captured Information about the time.

[0120] Backward hidden state ( ): Computed by backward LSTM units, capturing from the future Information about the time.

[0121] Final hidden state splicing formula:

[0122] ;

[0123] in, It is a forward and backward hidden state (dimension 256).

[0124] Transformer self-attention: strengthens the association of clinical terms (e.g., "pulmonary nodule-ground-glass opacity"), the formula is the same as SwingTransformer attention ( ), 6-layer encoder output context enhancement features.

[0125] Field-level attention: Focuses on high-risk information (such as smoking history), weight formula:

[0126] ;

[0127] in, For text field features, For learnable weights, The field weights are used to generate a 768-dimensional text feature vector.

[0128] 3. Multimodal fusion module

[0129] By employing a layered fusion strategy, deep synergy between image and text features is achieved:

[0130] By employing a hierarchical strategy of "mid-term interactive correlation - late-term high-order fusion," deep collaboration between CT image features and text data is achieved. Furthermore, by optimizing the Tucker decomposition bilinear fusion technique, the problems of parameter explosion and loss of high-order interactive information in traditional multimodal fusion are addressed.

[0131] (1) Cross-modal attention (CMA): such as Figure 4 As shown, image features are the primary driver, and textual information is explicitly associated with them. The formula is:

[0132] ;

[0133] in, Image features For text features, A modality-adapted weight matrix is ​​then used. Finally, this matrix is ​​fused with the original features to form interactive features, capturing cross-modal associations such as "nodular spiculation sign - smoking history".

[0134] (2) Tucker decomposition bilinear fusion: such as Figure 5 As shown, by using tensor low-rank decomposition, the parameter scale is reduced while preserving the high-order interaction between image and text features. The tensor decomposition formula is as follows:

[0135] ;

[0136] in, For low-dimensional core tensors ( (Encoded image-text core interaction mode). This is the image feature projection matrix (mapping high-dimensional image features to 64 dimensions). This is the text feature projection matrix (which also maps the text features to 64 dimensions). The output projection matrix maps the 64-dimensional fused features to the required dimensional space for the output. This is a tensor outer product operation.

[0137] (3) Multi-task loss function: Simultaneously optimize benign / malignant classification and subtype identification, formula:

[0138] ;

[0139] Focus loss ( (To resolve class imbalance)

[0140] ;

[0141] in, To predict probabilities for the model, (Weight of malignant samples) (Focusing on difficult samples).

[0142] Comparative learning regularization terms ( (Enhancing feature discriminativeness)

[0143] ;

[0144] in, For sample features, Features for positive / negative examples This represents the set of negative samples in the current batch. For cosine similarity, For temperature coefficient, , This is the loss weight.

[0145] 4. Diagnostic and Interpretable Output Module

[0146] The system outputs "probability of benign or malignant transformation + pathological subtype (adenocarcinoma / squamous carcinoma, etc.)" and enhances clinical credibility by visualizing key areas of interest in CT images (such as nodule boundaries) and ranking text keywords with attention weights (such as "elevated tumor marker CEA").

[0147] II. The detailed content of the lung cancer image and text-assisted analysis method based on multimodal fusion is as follows:

[0148] 1. Implementation basis;

[0149] Relying on the scientific research big data platform of the cooperating hospital, data of patients in the lung cancer disease database (including low-dose CT and electronic medical records) were collected, and were anonymized after ethical review (in accordance with the "Medical Data Security Specification").

[0150] 2. Detailed implementation steps;

[0151] (1) Data preprocessing: CT image standardization (WL=-600HU, WW=1500HU), lung field segmentation (Dice≥0.95), text entity recognition (F1≥0.9) and vectorization mapping of SNOMED CT terminology are completed according to the calculation formula of the multi-source data preprocessing module in the above system;

[0152] (2) Train the model in the single-modal deep learning feature extraction module; extract image features through the image feature extraction model and extract text features through the text feature extraction model;

[0153] 1) Image feature extraction model: AdamW optimizer, learning rate 1e-4, Transformer (window size 7×7), trained for 100 epochs;

[0154] 2) Text feature extraction model: BiLSTM (256-dimensional hidden layers) + 6 Transformer layers, learning rate 5e-5, optimized with cross-entropy loss;

[0155] (3) Multimodal fusion training: CMA attention head number 8, Tucker decomposition core tensor 64×64×64, combined loss function (λ1=0.7, λ2=0.3), 5-fold cross-validation optimization;

[0156] (5) Output analysis and diagnostic results;

[0157] (4) Clinical validation and deployment: Clinical validation will be conducted in partner hospitals to test the practicality of the model based on multiple actual samples. After physicians refer to the model output results, the false negative rate will be reduced and the accuracy of clinical diagnosis will be improved. The model will be packaged into an SDK and deployed to the PACS system of primary healthcare institutions using Docker containerization technology to ensure system compatibility and ease of operation and maintenance. At the same time, inference efficiency will be optimized to ensure that the analysis time of a single sample meets the needs of efficient clinical response and is suitable for practical applications in primary healthcare scenarios.

[0158] Technical effects of the present invention: Compared with the single-modal diagnostic model, the multimodal fusion model of the present invention significantly improves the diagnostic performance in the early screening task of lung cancer. The ability to distinguish between benign and malignant nodules and the accuracy of pathological subtype identification are greatly optimized. In particular, the ability to detect early small lesions is significantly enhanced, effectively making up for the diagnostic blind spots of single-modal data.

[0159] The clinical effects of this invention are: significantly shortening the diagnostic time and greatly improving efficiency compared to manual image reading; reducing the false positive rate and unnecessary follow-up examinations; assisting in the early detection of lung cancer, giving patients the best treatment opportunity, and playing an important role in improving patient survival rates.

[0160] The social benefits of this invention are as follows: The model supports lightweight deployment, can be adapted to the equipment conditions of primary healthcare institutions, alleviates the problem of uneven distribution of medical resources, and provides technical support for the goal of early cancer screening in the "Healthy China" strategy; at the same time, it reduces the waste of resources in the diagnosis and treatment of lung cancer and helps to control the overall national medical expenditure.

Claims

1. A lung cancer image and text-assisted analysis system based on multimodal fusion, characterized in that: The system includes a multi-source data preprocessing module for preprocessing lung cancer CT images and clinical texts, a single-modal deep learning feature extraction module for extracting features from lung cancer CT images and clinical texts, a multi-modal fusion module for fusing the features extracted from lung cancer CT images and clinical texts, and a diagnostic and interpretability output module for outputting analysis results and assisting in diagnosis. The signal output terminal of the multi-source data preprocessing module is connected to the signal input terminal of the single-modal deep learning feature extraction module, the signal output terminal of the single-modal deep learning feature extraction module is connected to the signal input terminal of the multi-modal fusion module, and the signal output terminal of the multi-modal fusion module is connected to the signal input terminal of the diagnostic and interpretability output module.

2. A method for lung cancer image and text-assisted analysis based on multimodal fusion, implemented by the lung cancer image and text-assisted analysis system based on multimodal fusion as described in claim 1; characterized in that: Includes the following steps: S1. Perform preprocessing operations on the patient's lung cancer CT images through the multi-source data preprocessing module; S2. Preprocess the patient's clinical text using the multi-source data preprocessing module; S3. The preprocessed lung cancer CT images are subjected to feature extraction operations through a single-modal deep learning feature extraction module to obtain image features; S4. The preprocessed clinical text is subjected to feature extraction operation through the single-modal deep learning feature extraction module to obtain text features; S5. The image features and text features are fused using the multimodal fusion module to obtain fused features; S6. Input the fused features into the diagnostic and interpretability output module, which then analyzes and outputs the diagnostic results.

3. The lung cancer image and text-assisted analysis method based on multimodal fusion as described in claim 2, characterized in that: Step S1 specifically includes the following steps: S11. Standardize the grayscale range of lung cancer CT images. The calculation formula is as follows: ； in, The original lung cancer CT image shows voxel gray values. For lung window position, For the width of the lung window, This means restricting the result within the parentheses to the interval [0,1]. The standardized grayscale value; S12. An improved U-Net architecture is used for lung field segmentation, and the Dice loss function is used to optimize the segmentation accuracy. The calculation formula is as follows: ； in, To predict lung field regions for the model, To establish the gold standard for physicians; S13. Resample the original lung cancer CT images using linear interpolation. Voxels; samples are expanded using elastic deformation, and the formula for calculating elastic deformation is: ； in, It is a Gaussian random displacement field.

4. The lung cancer image and text-assisted analysis method based on multimodal fusion as described in claim 3, characterized in that: Step S2 specifically includes the following steps: S21. Entities are extracted from clinical text based on BERT-like models. The output layer incorporates a conditional random field (CRF) to optimize entity boundaries. The loss function of the CRF is: ； in, For state transition weights, The entity label score output by BioBERT; S22. The extracted entities are mapped to SNOMED CT term codes to achieve term standardization; then, the standardized term codes are transformed into a high-dimensional semantic feature vector sequence through a word embedding layer.

5. The lung cancer image and text-assisted analysis method based on multimodal fusion as described in claim 4, characterized in that: Step S3 specifically includes the following steps: S31. Construct an image feature extraction model to perform feature extraction operations on the preprocessed lung cancer CT images to obtain image features; the image feature extraction model consists of residual blocks, window attention modules, channel attention modules, and spatial attention modules. S32. Residual blocks alleviate gradient vanishing through cross-layer connections; the calculation formula is as follows: ； in, Dimension matching implemented for 1×1×1 convolutions; S33. A feature extraction network based on the Transformer architecture extracts global and local features in stages. The calculation formula is as follows: ； in, The feature matrix within the window. This is the relative position offset matrix; S34. The channel attention module adaptively adjusts the channel weights, and its calculation formula is as follows: ； in, This is the result of global average pooling. For the weights of the fully connected layer, For the Sigmoid function; S35. The spatial attention module generates a spatial weight map, and its calculation formula is as follows: ； in, The spatial weight map is the output of a 7×7 convolution.

6. The lung cancer image and text-assisted analysis method based on multimodal fusion as described in claim 5, characterized in that: Step S4 specifically includes the following steps: S41. A bidirectional long short-term memory network is used to process the high-dimensional semantic feature vector sequence, capturing long-range contextual dependencies between clinical terms, and memorizing sequence information through hidden concatenation; the calculation formula is as follows: ； in, It is in a forward and backward hidden state; S42. Use the window attention module to enhance the association of clinical terms; S43. Focusing on high-risk information, the field weights are calculated using the following formula: ； in, For text field features, For learnable weights, Assign field weights; S44. Generate text features by weighting the 512-dimensional text feature vector according to the field weights.

7. The lung cancer image and text-assisted analysis method based on multimodal fusion as described in claim 6, characterized in that: Step S5 specifically includes the following steps: S51. Taking image features as the primary factor and explicitly associating them with text features, the calculation formula is as follows: ； in, Image features For text features, Modal adaptation weight matrix; S52. By using tensor low-rank decomposition, the parameter scale is reduced while preserving the high-order interaction between image features and text features. The calculation formula is as follows: ； in, For low-dimensional core tensors, For image feature projection matrix, The text feature projection matrix, To output the projection matrix, This is a tensor outer product operation; S53. Simultaneously optimize benign / malignant classification and subtype identification; the calculation formula is as follows: ； S54. Calculate the focus loss. The formula is as follows: ； in, Predict probabilities for the model; S55. Contrastive learning regularization term, which enhances feature discriminative power, is calculated using the following formula: ； in, For sample features, Features for positive / negative examples This represents the set of negative samples in the current batch. For cosine similarity, This is the temperature coefficient.

8. The lung cancer image and text-assisted analysis method based on multimodal fusion as described in claim 7, characterized in that: In step S6, the diagnosis and interpretability output module outputs the probability of benign or malignant transformation and the pathological subtype, and visualizes the key areas of lung cancer CT images through Grad-CAM. At the same time, it sorts the keywords in the clinical text according to attention weight to improve clinical credibility.

Images