A smart lung cancer detection system based on PET / MR multimodal imaging

Abstract

This invention relates to the field of intelligent medicine, specifically to an intelligent lung cancer detection system based on PET / MR multimodal imaging, comprising a data acquisition module, a data preprocessing module, an intelligent multimodal classification module, and a lung cancer detection module. The invention employs a cross-modal feature fusion method based on a bidirectional attention mechanism. Building upon multimodal fusion, it considers fine-grained visual features through visual embedding and integrates single-modal features to enhance visual modality features. Simultaneously, it performs object-level semantic mining of lung cancer pathology, considering both cross-modal interaction and emphasizing the importance of a single modality, thus achieving accurate lung condition detection. Furthermore, the invention utilizes a multi-scale hybrid UNet model based on cross-channel fusion to segment lung cancer lesion regions in multimodal fused images, enhancing semantic information extraction capabilities and cross-modal fusion performance, promoting multi-scale information fusion, enhancing feature representation capabilities at different scales, and increasing the accuracy of lung cancer detection.

Description

Technical Field

[0001] This invention belongs to the field of intelligent medicine, specifically a lung cancer intelligent detection system based on PET / MR multimodal imaging. Background Technology

[0002] Lung cancer is one of the most common cancers worldwide and a leading cause of cancer death. Early detection and diagnosis of lung cancer are crucial for improving treatment outcomes and survival rates. However, existing lung cancer detection methods are usually single-modal, which means that lung cancer lesions may not be easily detected in certain image modalities, thus limiting the accuracy of detection. Summary of the Invention

[0003] To address the above issues and overcome the shortcomings of existing technologies, this invention provides an intelligent lung cancer detection system based on PET / MR multimodal imaging. Addressing the problem that existing lung cancer detection methods are typically single-modal and thus have low accuracy, this invention creatively employs a cross-modal feature fusion method based on a bidirectional attention mechanism. Building upon multimodal fusion of PET and MR images, it enhances visual modality features by considering fine-grained visual features through visual embedding and integrating single-modal features. Simultaneously, it performs object-level semantic mining of lung cancer pathology and conducts modal interaction based on an attention mechanism for semantic features and global-local visual features. This invention considers both cross-modal interactions and emphasizes the importance of each individual modality, thereby achieving cross-modal feature fusion and accurate lung condition detection. It creatively employs a multi-scale hybrid UNet model based on cross-channel fusion to segment lung cancer lesion regions in multi-modal fused images. This effectively extracts channel information from the multi-scale encoder, capturing rich global features along the channels. This enhances the long-range semantic information extraction capability and cross-modal fusion performance of the bidirectional attention-based cross-modal feature fusion method, promotes multi-scale information fusion, strengthens feature representation capabilities at different scales, and further increases the accuracy of lung cancer detection.

[0004] The present invention provides an intelligent lung cancer detection system based on PET / MR multimodal imaging, comprising a data acquisition module, a data preprocessing module, an intelligent multimodal classification module, and a lung cancer detection module;

[0005] The data acquisition module uses PET images, MR images, and pathological data of the lungs of patients in the respiratory medicine department.

[0006] The data preprocessing module performs noise reduction processing on PET and MR images;

[0007] The intelligent multimodal classification module registers and fuses PET and MR images to obtain a fused image. It uses a multi-scale hybrid UNet model based on cross-channel fusion to segment the lesion region of the fused image to obtain a segmented image. It uses a cross-modal feature fusion method based on bidirectional attention mechanism to perform cross-modal fusion and prediction classification of pathological data and segmented images to obtain lung condition classification results.

[0008] The lung cancer detection module uses PET images, MR images, and pathological data of the lungs of respiratory medicine patients with known lung cancer conditions to train an intelligent multimodal classification module. The trained intelligent multimodal classification module then classifies and detects the lung condition of respiratory medicine patients with unknown lung cancer conditions, thus obtaining lung cancer detection results.

[0009] Furthermore, in the intelligent multimodal classification module, the calculation process for the lung condition classification result specifically includes the following steps:

[0010] Step S1: Semantic feature extraction, using the Sentence-BERT model to extract semantic features from lung pathology data;

[0011] Step S2: Image-level multimodal fusion. The PET and MR images are fused based on location registration to obtain a fused image. This includes the following steps:

[0012] Step S21: Image preprocessing. Preprocess the PET image and MR image to ensure that the two images have the same spatial resolution and coordinate system.

[0013] Step S22: Position registration, register the PET image and the MR image so that they are aligned in the same coordinate system;

[0014] Step S23: Image-level fusion, weighted summation of the registered PET image and MR image to obtain the fused image;

[0015] Step S3: Image fusion segmentation. A multi-scale hybrid UNet model based on cross-channel fusion is used to segment the fused image to obtain a segmented image. This includes the following steps:

[0016] Step S31: Multi-layer downsampling. The encoder is used to perform feature encoding and four-layer downsampling on the fused image to obtain the initial encoded features and the four-layer downsampled features.

[0017] Step S32: Reshape and map the downsampled features of the second and third layers respectively, so that their dimensions are consistent with the downsampled features of the first layer, to obtain the reshaped features:

[0018] ;

[0019] In the formula, Represents the downsampling features of the second and third layers. Represents the deconvolution function. Represents the convolution operation. The remodeling features are represented by the remodeling features of the second and third layers, which are consistent with the downsampling features of the first layer.

[0020] Step S32: Concatenate the reshaped features with the downsampling from the first layer to obtain the combined features;

[0021] Step S33: Divide the combined features into multiple patches, and perform feature fusion along the channel dimension on the multiple patches to obtain the global fused features:

[0022] ;

[0023] In the formula, Representative combination characteristics, Representative level normalization, Represents channel mixing operation. Represents global fusion characteristics;

[0024] Step S34: Project the global fused features into two-dimensional features, and split the two-dimensional features into three independent features on an average basis;

[0025] Step S35: Reshape and map the three independent features respectively so that their dimensions are consistent with the downsampled features of the first to third layers, thus obtaining the three-layer optimized downsampled features;

[0026] Step S36: Feature upsampling. The initial encoded features, the three-layer optimized downsampled features, and the fourth-layer downsampled features are used as input to the decoder for upsampling to obtain the segmentation feature map.

[0027] Step S37: Activate the segmentation feature map to obtain the segmentation image;

[0028] Step S4: Cross-modal feature fusion. The segmented image undergoes embedding processing and feature extraction to obtain segmented image features. These segmented image features and semantic features are then fused using a bidirectional attention mechanism and a softmax function to predict the output, yielding a comprehensive predicted value. This includes the following steps:

[0029] Step S41: Graph embedding processing, which divides the segmented image into n regions on average, and each region is further divided into n smaller blocks;

[0030] Step S42: Linear projection, flatten all the small patches of each region, and project all the small patches of each region into d dimensions through a learnable linear projection matrix to obtain all the embedding representations of each region;

[0031] Step S43: Aggregate all embeddings for each region and add tagged embeddings and positional embeddings to obtain the visual embedding for each region:

[0032] Z i =[ z c ; z 1 ,..., z n ]+E ;

[0033] In the formula, This indicates the added tag embedding. Represents the first embedding representation. This indicates the added positional embeddings, used to specify the relative positions of all embedded representations within the region. The visual embedding representing the i-th region;

[0034] Step S44: Input all visual embeddings into the encoder to obtain visual region features, and combine all visual region features to obtain segmented image features;

[0035] Step S45: Cross-modal feature fusion based on bidirectional attention mechanism. Semantic features and segmented image features are fused using a bidirectional attention mechanism and the softmax function is used for prediction to obtain a comprehensive predicted value. This includes the following steps:

[0036] Step S451: Text guidance feature extraction based on multi-head cross-attention mechanism. A multi-head cross-attention mechanism is used to extract cross-modal features from semantic features and segmented image features to obtain text guidance features:

[0037] ;

[0038] In the formula, Representative level normalization, Represents semantic features, Represents segmentation image features, This represents the processing of multi-head cross-attention mechanisms. Represents textual guidance features;

[0039] Step S452: Feature optimization based on multi-head self-attention mechanism. The text guidance features are optimized using a multi-head self-attention mechanism to obtain the optimized text guidance features:

[0040] ;

[0041] In the formula, Represents multi-head self-attention mechanism processing, This represents optimized text guidance features;

[0042] Step S453: The feedforward neural network output is used to perform layer normalization on the optimized text guidance features and the feedforward neural network output to obtain the cross-modal features of the text guidance.

[0043] ;

[0044] In the formula, Representation layer normalization, This indicates that the feedforward neural network is being processed. Cross-modal features representing text guidance;

[0045] Step S454: Visually guided cross-modal feature calculation. Repeat steps S451-S453, using the segmented image features as input, to obtain the visually guided cross-modal features:

[0046] ;

[0047] ;

[0048] ;

[0049] Step S455: Cross-modal feature fusion, fusing text-guided cross-modal features with visual-guided cross-modal features to obtain bidirectional cross-modal features;

[0050] Step S456: Multimodal feature fusion output. A multi-head self-attention mechanism is used to fuse semantic features, segmentation image features, and bidirectional cross-modal features, and weights are added for adjustment to obtain the multimodal output:

[0051] y m =softmax( W m MSA([ h T , h I ,g])+b) ;

[0052] In the formula, Indicates bidirectional cross-modal features, To preset weights, To adjust the parameters, Represents multimodal output;

[0053] Step S457: Multimodal feature synthesis output. Weighted activation outputs are applied to semantic features and segmented image features respectively, and then added to the multimodal output to obtain the comprehensive prediction value.

[0054] ;

[0055] ;

[0056] ;

[0057] In the formula, , To preset weights, To adjust the parameters, and These represent the text prediction value and the visual prediction value, respectively. This represents the overall forecast value;

[0058] Step S5: Lung condition classification. Based on the comprehensive prediction value, the lung condition of all patients is classified to obtain the lung condition classification results.

[0059] The beneficial results achieved by the present invention using the above solution are as follows:

[0060] (1) In view of the problem that existing lung cancer detection methods are usually single-modal detection and thus have low accuracy, this invention creatively adopts a cross-modal feature fusion method based on bidirectional attention mechanism. On the basis of multimodal fusion of PET images and MR, fine-grained visual features are considered through visual embedding and single-modal features are integrated to enhance the features of visual modality. At the same time, object-level semantic mining of lung cancer pathology is performed, and modal interaction based on attention mechanism is performed on semantic features and global-local visual features. It considers cross-modal interaction and emphasizes the importance of the single modality, thereby realizing the fusion of cross-modal features and accurate lung condition detection.

[0061] (2) This invention creatively employs a multi-scale hybrid UNet model based on cross-channel fusion to segment the lung cancer lesion region of the multimodal fusion image, effectively extracts channel information from the multi-scale encoder, and enhances the ability of the cross-modal feature fusion method based on bidirectional attention mechanism to extract long-range semantic information and cross-modal fusion performance by capturing rich global features along the channel. This promotes multi-scale information fusion, enhances the feature representation ability at different scales, and further increases the accuracy of lung cancer detection. Attached Figure Description

[0062] Figure 1 A block diagram of a lung cancer intelligent detection system based on PET / MR multimodal imaging provided by the present invention;

[0063] Figure 2 This is a flowchart illustrating a cross-modal feature fusion method based on a bidirectional attention mechanism.

[0064] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with the embodiments of the invention to explain the invention and do not constitute a limitation thereof. Detailed Implementation

[0065] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0066] Example 1, see Figure 1 The present invention provides an intelligent lung cancer detection system based on PET / MR multimodal imaging, including a data acquisition module, a data preprocessing module, an intelligent multimodal classification module, and a lung cancer detection module;

[0067] The data acquisition module uses PET images, MR images, and pathological data of the lungs of patients in the respiratory medicine department.

[0068] The data preprocessing module performs noise reduction processing on PET and MR images;

[0069] The intelligent multimodal classification module registers and fuses PET and MR images to obtain a fused image. It uses a multi-scale hybrid UNet model based on cross-channel fusion to segment the lesion region of the fused image to obtain a segmented image. It uses a cross-modal feature fusion method based on bidirectional attention mechanism to perform cross-modal fusion and prediction classification of pathological data and segmented images to obtain lung condition classification results.

[0070] The lung cancer detection module uses PET images, MR images, and pathological data of the lungs of respiratory medicine patients with known lung cancer conditions to train an intelligent multimodal classification module. The trained intelligent multimodal classification module then classifies and detects the lung condition of respiratory medicine patients with unknown lung cancer conditions, thus obtaining lung cancer detection results.

[0071] Example 2: Based on the above examples, the calculation process of the lung condition classification result in the intelligent multimodal classification module specifically includes the following steps:

[0072] Step S1: Semantic feature extraction, using the Sentence-BERT model to extract semantic features from lung pathology data;

[0073] Step S2: Image-level multimodal fusion, performing image-level fusion based on location registration on PET and MR images to obtain a fused image;

[0074] Step S3: Image fusion segmentation. A multi-scale hybrid UNet model based on cross-channel fusion is used to segment the fused image to obtain a segmented image.

[0075] Step S4: Cross-modal feature fusion. Embedding and feature extraction are performed on the segmented image to obtain segmented image features. Cross-modal feature fusion based on bidirectional attention mechanism and softmax function are performed on the segmented image features and semantic features to obtain a comprehensive prediction value.

[0076] Step S5: Lung condition classification. Based on the comprehensive prediction value, the lung condition of all patients is classified to obtain the lung condition classification results.

[0077] Example 3, this example is based on the above example, step S2 specifically includes the following steps:

[0078] Step S21: Image preprocessing. Preprocess the PET image and MR image to ensure that the two images have the same spatial resolution and coordinate system.

[0079] Step S22: Position registration, register the PET image and the MR image so that they are aligned in the same coordinate system;

[0080] Step S23: Image-level fusion, weighted summation of the registered PET image and MR image to obtain the fused image.

[0081] Example 4: This example is based on the above examples. Step S3 specifically includes the following steps:

[0082] Step S31: Multi-layer downsampling. The encoder is used to perform feature encoding and four-layer downsampling on the fused image to obtain the initial encoded features and the four-layer downsampled features.

[0083] Step S32: Reshape and map the downsampled features of the second and third layers respectively, so that their dimensions are consistent with the downsampled features of the first layer, to obtain the reshaped features.

[0084] ;

[0085] In the formula, Represents the downsampling features of the second and third layers. Represents the deconvolution function. Represents the convolution operation. The remodeling features are represented by the remodeling features of the second and third layers, which are consistent with the downsampling features of the first layer.

[0086] Step S32: Concatenate the reshaped features with the downsampling from the first layer to obtain the combined features;

[0087] Step S33: Divide the combined features into multiple patches, and perform feature fusion along the channel dimension on the multiple patches to obtain the global fused features:

[0088] ;

[0089] In the formula, Representative combination characteristics, Representative level normalization, Represents channel mixing operation. Represents global fusion characteristics;

[0090] Step S34: Project the global fused features into two-dimensional features, and split the two-dimensional features into three independent features on an average basis;

[0091] Step S35: Reshape and map the three independent features respectively so that their dimensions are consistent with the downsampled features of the first to third layers, thus obtaining the three-layer optimized downsampled features;

[0092] Step S36: Feature upsampling. The initial encoded features, the three-layer optimized downsampled features, and the fourth-layer downsampled features are used as input to the decoder for upsampling to obtain the segmentation feature map.

[0093] Step S37: Activate the segmentation feature map to obtain the segmented image.

[0094] By performing the above operations, this invention creatively employs a multi-scale hybrid UNet model based on cross-channel fusion to segment the lung cancer lesion region in a multimodal fusion image. It effectively extracts channel information from the multi-scale encoder, and by capturing rich global features along the channels, it enhances the ability of the cross-modal feature fusion method based on bidirectional attention mechanism to extract long-range semantic information and improves the cross-modal fusion performance. This promotes multi-scale information fusion, enhances the feature representation ability at different scales, and further increases the accuracy of lung cancer detection.

[0095] Example 5, see Figure 2 This embodiment is based on the above embodiment. Step S4 specifically includes the following steps:

[0096] Step S41: Graph embedding processing, which divides the segmented image into n regions on average, and each region is further divided into n smaller blocks;

[0097] Step S42: Linear projection, flatten all the small patches of each region, and project all the small patches of each region into d dimensions through a learnable linear projection matrix to obtain all the embedding representations of each region;

[0098] Step S43: Aggregate all embeddings for each region and add tagged embeddings and positional embeddings to obtain the visual embedding for each region:

[0099] Z i =[ z c ; z 1 ,..., z n ]+E ;

[0100] In the formula, This indicates the added tag embedding. Represents the first embedding representation. This indicates the added positional embeddings, used to specify the relative positions of all embedded representations within the region. Visual embedding representing the i-th region;

[0101] Step S44: Input all visual embeddings into the encoder to obtain visual region features, and combine all visual region features to obtain segmented image features;

[0102] Step S45: Cross-modal feature fusion based on bidirectional attention mechanism: Perform cross-modal feature fusion based on bidirectional attention mechanism and softmax function prediction output on semantic features and segmented image features to obtain comprehensive prediction value.

[0103] Example 6, this example is based on the above example, step S45 specifically includes the following steps:

[0104] Step S451: Text guidance feature extraction based on multi-head cross-attention mechanism. A multi-head cross-attention mechanism is used to extract cross-modal features from semantic features and segmented image features to obtain text guidance features:

[0105] ;

[0106] In the formula, Representative level normalization, Represents semantic features, Represents segmentation image features, This represents the processing of multi-head cross-attention mechanisms. Represents textual guidance features;

[0107] Step S452: Feature optimization based on multi-head self-attention mechanism. The text guidance features are optimized using a multi-head self-attention mechanism to obtain the optimized text guidance features:

[0108] ;

[0109] In the formula, Represents multi-head self-attention mechanism processing, This represents optimized text guidance features;

[0110] Step S453: The feedforward neural network output is used to perform layer normalization on the optimized text guidance features and the feedforward neural network output to obtain the cross-modal features of the text guidance.

[0111] ;

[0112] In the formula, Representation layer normalization, This indicates that the feedforward neural network is being processed. Cross-modal features representing text guidance;

[0113] Step S454: Visually guided cross-modal feature calculation. Repeat steps S451-S453, using segmented image features as input, to obtain visually guided cross-modal features.

[0114] Step S455: Cross-modal feature fusion, fusing text-guided cross-modal features with visual-guided cross-modal features to obtain bidirectional cross-modal features;

[0115] Step S456: Multimodal feature fusion output. A multi-head self-attention mechanism is used to fuse semantic features, segmentation image features, and bidirectional cross-modal features, and weights are added for adjustment to obtain the multimodal output:

[0116] y m =softmax( W m MSA([ h T , h I ,g])+b) ;

[0117] In the formula, Indicates bidirectional cross-modal features, To preset weights, To adjust the parameters, Represents multimodal output;

[0118] Step S457: Multimodal feature synthesis output. Weighted activation outputs are applied to semantic features and segmented image features respectively, and then added to the multimodal output to obtain the predicted value.

[0119] ;

[0120] ;

[0121] ;

[0122] In the formula, , To preset weights, To adjust the parameters, and These represent the text prediction value and the visual prediction value, respectively. This represents the overall forecast value.

[0123] Example 7: This example is based on the above examples, using lung pathology data, and specifically includes the following details:

[0124] Histological and pathological data:

[0125] 1. Immunohistochemical data, including tumor markers and inflammatory markers;

[0126] 2. Molecular pathological data, including lung cancer-related gene mutations, RNA expression, or DNA methylation;

[0127] Respiratory function data:

[0128] 1. Pulmonary function test data, including forced expiratory volume in one second, forced vital capacity, and DLCO diffusion capacity;

[0129] 2. Dynamic respiratory monitoring data, including respiratory rate, gas exchange volume per breath, and minute ventilation;

[0130] Blood and body fluid pathology data:

[0131] 1. Blood gas analysis, including arterial oxygen partial pressure, arterial carbon dioxide partial pressure, and pH value;

[0132] 2. Inflammatory markers: C-reactive protein (CRP), white blood cell count, and cytokines;

[0133] 3. Lung cancer biomarkers, including CEA (carcinoembryonic antigen), CYFRA 21-1 non-small cell lung cancer detection, and NSE (neuron-specific enolase);

[0134] 4. Body fluid analysis, including pleural effusion analysis and cytological examination;

[0135] Symptom and sign data:

[0136] 1. Subjective symptoms, including cough type, dyspnea, chest pain, and hemoptysis;

[0137] Medical examination data:

[0138] 1. Percussion to determine if there is fluid accumulation or consolidation in the lungs;

[0139] 2. Auscultate to check for rales, wheezing, or chest friction rubs.

[0140] Example 8: Based on the above examples, in step S2, image-level fusion based on feature extraction can be optimized to perform image-level fusion on PET images and MR images. The optimized steps are as follows:

[0141] Image preprocessing:

[0142] Preprocessing is performed on PET and MR images, including denoising and grayscale normalization, to ensure that the two images have the same spatial resolution and coordinate system;

[0143] Registration:

[0144] The PET and MR images were registered using a local feature matching method.

[0145] Feature extraction:

[0146] Features, including texture and shape features, are extracted from PET and MR images to better represent image information;

[0147] Feature fusion: The features of PET and MR are fused using a deep learning model;

[0148] Complementary fusion: Utilizing the complementary information from PET and MR images, with PET providing metabolic information and MR providing anatomical information, information fusion is performed;

[0149] Post-processing:

[0150] The fused image is post-processed, including noise removal and smoothing, to obtain a clearer and more visually appealing fused image.

[0151] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0152] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

[0153] The present invention and its embodiments have been described above. This description is not restrictive, and the accompanying drawings are only one embodiment of the present invention; the actual structure is not limited thereto. In conclusion, if those skilled in the art are inspired by this description and design similar structures and embodiments without departing from the spirit of the invention, such designs should fall within the protection scope of the present invention.

Claims

1. A lung cancer intelligent detection system based on PET / MR multimodal imaging, characterized in that: It includes a data acquisition module, a data preprocessing module, an intelligent multimodal classification module, and a lung cancer detection module; The data acquisition module uses PET images, MR images, and lung pathology data of patients in the respiratory medicine department. The data preprocessing module performs noise reduction processing on PET and MR images; The intelligent multimodal classification module registers and fuses PET and MR images to obtain a fused image. It uses a multi-scale hybrid UNet model based on cross-channel fusion to segment the lesion region of the fused image to obtain a segmented image. It uses a cross-modal feature fusion method based on bidirectional attention mechanism to perform cross-modal fusion and prediction classification of pathological data and segmented images to obtain lung condition classification results. The calculation process for the lung condition classification results specifically includes the following steps: Step S1: Semantic feature extraction, using the Sentence-BERT model to extract semantic features from lung pathology data; Step S2: Image-level multimodal fusion, performing image-level fusion based on location registration on PET and MR images to obtain a fused image; Step S3: Image fusion segmentation. A multi-scale hybrid UNet model based on cross-channel fusion is used to segment the fused image to obtain a segmented image. This includes the following steps: Step S31: Multi-layer downsampling. The encoder is used to perform feature encoding and four-layer downsampling on the fused image to obtain the initial encoded features and the four-layer downsampled features. Step S32: Reshape and map the downsampled features of the second and third layers respectively, so that their dimensions are consistent with the downsampled features of the first layer, and obtain the reshaped features; Step S32: Concatenate the reshaped features with the downsampling from the first layer to obtain the combined features; Step S33: Divide the combined features into multiple patches, and perform feature fusion on the channel dimension of the multiple patches to obtain the global fused features; Step S34: Project the global fused features into two-dimensional features, and split the two-dimensional features into three independent features on an average basis; Step S35: Reshape and map the three independent features respectively so that their dimensions are consistent with the downsampled features of the first to third layers, thus obtaining the three-layer optimized downsampled features; Step S36: Feature upsampling. The initial encoded features, the three-layer optimized downsampled features, and the fourth-layer downsampled features are used as input to the decoder for upsampling to obtain the segmentation feature map. Step S37: Activate the segmentation feature map to obtain the segmentation image; Step S4: Cross-modal feature fusion. The segmented image undergoes embedding processing and feature extraction to obtain segmented image features. These features are then fused with semantic features using a bidirectional attention mechanism and a softmax function to predict the output, yielding a comprehensive predicted value. This includes the following steps: Step S41: Graph embedding processing, which divides the segmented image into n regions on average, and each region is further divided into n smaller blocks; Step S42: Linear projection, flatten all the small patches of each region, and project all the small patches of each region into d dimensions through a learnable linear projection matrix to obtain all the embedding representations of each region; Step S43: Aggregate all embedding representations for each region and add labeled embeddings and positional embeddings to obtain the visual embedding for each region: ； In the formula, This indicates the added tag embedding. Represents the first embedding representation. This indicates the added positional embeddings, used to specify the relative positions of all embedded representations within the region. The visual embedding representing the i-th region; Step S44: Input all visual embeddings into the encoder for feature encoding to obtain visual region features, and combine all visual region features to obtain segmented image features; Step S45: Cross-modal feature fusion based on bidirectional attention mechanism. Semantic features and segmented image features are fused using a bidirectional attention mechanism and the softmax function is used for prediction to obtain a comprehensive predicted value. This includes the following steps: Step S451: Text guidance feature extraction based on multi-head cross-attention mechanism. A multi-head cross-attention mechanism is used to extract cross-modal features from semantic features and segmented image features to obtain text guidance features: ； In the formula, Representative level normalization, Represents semantic features, Represents segmentation image features, This represents the processing of multi-head cross-attention mechanisms. Represents textual guidance features; Step S452: Feature optimization based on multi-head self-attention mechanism. The text guidance features are optimized using a multi-head self-attention mechanism to obtain the optimized text guidance features: ； In the formula, Represents multi-head self-attention mechanism processing, This represents optimized text guidance features; Step S453: The feedforward neural network output is used to perform layer normalization on the optimized text guidance features and the feedforward neural network output to obtain the cross-modal features of the text guidance. ； In the formula, Representation layer normalization, This indicates feedforward neural network processing. Cross-modal features representing text guidance; Step S454: Visually guided cross-modal feature calculation. Repeat steps S451-S453, using the segmented image features as input, to obtain the visually guided cross-modal features: ； ； ； Step S455: Cross-modal feature fusion, fusing text-guided cross-modal features with visual-guided cross-modal features to obtain bidirectional cross-modal features; Step S456: Multimodal feature fusion output. A multi-head self-attention mechanism is used to fuse semantic features, segmentation image features, and bidirectional cross-modal features, and weights are added for adjustment to obtain the multimodal output: ； In the formula, Indicates bidirectional cross-modal features, To preset weights, To adjust the parameters, Represents multimodal output; Step S457: Multimodal feature synthesis output. Weighted activation outputs are applied to semantic features and segmented image features respectively, and then added to the multimodal output to obtain the comprehensive prediction value. ； ； ； In the formula, , To preset weights, To adjust the parameters, and These represent the text prediction value and the visual prediction value, respectively. This represents the overall forecast value; Step S5: Lung condition classification. Based on the comprehensive prediction value, the lung condition of all patients is classified to obtain the lung condition classification results. The lung cancer detection module uses PET images, MR images, and lung pathology data of respiratory medicine patients with known lung cancer conditions to train an intelligent multimodal classification module. The trained intelligent multimodal classification module then classifies and detects the lung condition of respiratory medicine patients with unknown lung cancer conditions, thus obtaining lung cancer detection results.

Images