Intelligent lung injury detection system based on lung imaging and pathological data

Abstract

The present invention relates to the field of intelligent medicine, and specifically to an intelligent lung injury detection system based on lung images and pathological data, comprising a multimodal data acquisition module, an intelligent segmentation module, a cross-modal feature fusion module, and a lung injury classification and detection module. The present invention adopts a cross-attention multimodal feature fusion method based on bidirectional gating to encode lung CT images and pathological text, uses a bidirectional gated attention mechanism to achieve modal interaction to highlight the lesion area, introduces a channel-space attention gate, optimizes feature fusion, and achieves accurate lung injury detection. A medical image segmentation method based on multi-layer fusion optimization and multi-layer parallel enhancement is adopted. Through multi-layer feature optimization based on depthwise separable convolution and three-layer parallel feature enhancement based on a residual mechanism, the key feature, namely the lesion area, is retained, nonlinear information is captured, and local-global features are effectively integrated to accurately segment the lung lesion area, thereby increasing the accuracy of lung injury detection.

Description

Technical Field

[0001] The present invention belongs to the field of intelligent medicine, and specifically relates to an intelligent lung injury detection system based on lung images and pathological data. Background Art

[0002] Lung injury detection is an important application in the medical field. It mainly analyzes and diagnoses lung imaging data and clinical pathology data, aiming to detect and evaluate lung injuries, lesions or diseases early. Existing artificial intelligence technology has difficulty in effectively and accurately detecting lung injuries when faced with lung imaging and clinical pathology data of pulmonary patients. Summary of the Invention

[0003] In view of the above situation, in order to overcome the defects of the existing technology, the present invention provides an intelligent lung injury detection system based on lung images and pathological data. In view of the problem that the existing artificial intelligence technology is difficult to effectively and accurately detect lung injury when facing lung images and clinical pathological data of pulmonary patients, the present invention creatively adopts a cross-attention multimodal feature fusion method based on bidirectional gating, adopts 3D Swin Transformer extracts local and global features of lung CT images, retains the spatial hierarchical structure, uses BioClinicalBERT to dynamically semantically encode pathological text, captures medical entities and time series features in symptom descriptions, and realizes modal interaction through a bidirectional gated attention mechanism. It locates the text semantic focus corresponding to the image features and enhances the visual salience of the lesion area. It introduces a channel-spatial attention gate to dynamically adjust the fusion weights of multimodal features, ultimately achieving optimal feature fusion and activation classification, and realizing accurate lung injury detection. The present invention creatively adopts a medical image segmentation method based on multi-layer fusion optimization and multi-layer parallel enhancement to segment lung CT images. Through multi-layer feature optimization based on depthwise separable convolution, it effectively extracts detailed local features and global context information of lung CT. Through three-layer parallel feature enhancement based on the residual mechanism, it retains the key feature, namely the lesion area, and effectively integrates local-global features while capturing nonlinear information. Finally, on this basis, the lung lesion area is accurately segmented, thereby optimizing cross-modal feature fusion and further improving the accuracy of lung injury detection.

[0004] The intelligent lung injury detection system based on lung images and pathological data provided by the present invention includes a multimodal data acquisition module, an intelligent segmentation module, a cross-modal feature fusion module and a lung injury classification and detection module;

[0005] The multimodal data acquisition module collects lung CT images from the lung medical imaging database and collects pathological data of patients visiting the pulmonary disease department;

[0006] The intelligent segmentation module uses a medical image segmentation method based on multi-layer fusion optimization and multi-layer parallel enhancement to segment the lung CT image to obtain a lesion image;

[0007] The cross-modal feature fusion module adopts a cross-attention multimodal feature fusion method based on bidirectional gating to perform feature fusion extraction on pathological data and lesion images to obtain cross-modal fusion features;

[0008] The lung injury classification detection module activates and classifies the fused features to obtain a lung injury classification detection result.

[0009] Furthermore, in the intelligent segmentation module, a medical image segmentation method based on multi-layer fusion optimization and multi-layer parallel enhancement specifically includes the following steps:

[0010] Step S1: Multi-layer downsampling: Input the lung CT image into a pre-trained three-layer encoder network for downsampling to obtain three-level encoding features 、 、 ;

[0011] Step S2: Optimize the attention module based on position embedding to encode features Optimize the attention module based on position embedding to obtain refined features, which includes the following steps:

[0012] Step S21: Feature position encoding embedding, encoding features Perform position encoding to obtain position encoding features;

[0013] Step S22: Matrix mapping, mapping and embedding the position encoding features to obtain the query matrix Q, key matrix K and value matrix V;

[0014] Step S23: self-attention mechanism processing, performing self-attention processing on the query matrix Q, key matrix K and value matrix V to obtain self-attention features;

[0015] Step S24: Feature fusion, encoding features Connect with the self-attention feature to obtain the refined feature ;

[0016] Step S3: Multi-layer feature optimization based on depth-wise separable convolution to refine features Perform multi-layer feature optimization based on depth-separable convolution to obtain optimized features :

[0017] ;

[0018] Where, stands for batch normalization, represents the ReLU activation function, represents a depthwise separable convolution with a kernel size of k, where k=2i-1, represents the standard convolution with a kernel size of k×k, Representative optimization features ;

[0019] Step S4: Three-layer parallel feature enhancement based on residual mechanism to enhance the encoding features Perform three-layer parallel feature enhancement based on the residual mechanism to obtain enhanced features , specifically including the following steps:

[0020] Step S41: Standard convolution and activation layer processing, encoding features Perform standard convolution, batch normalization and ReLU activation to obtain standard features :

[0021] ;

[0022] Where, Representative standard features ;

[0023] Step S42: Based on the local-global feature integration layer processing of the dual-pooling, the standard feature Perform maximum pooling and average pooling respectively, and concatenate the results, perform standard convolution, batch normalization and ReLU activation to obtain enhanced features :

[0024] i 2 =ReLU(BN(Con v 3 [ p 3 avg ( i 1 )⨂ p 3 max ( i 1 )])) ;

[0025] Where, Represents the average pooling process with a convolution kernel size of 3×3, Represents the maximum pooling process with a convolution kernel size of 3×3, Represents a connection operation, Represents enhanced features ;

[0026] Step S43: Global pooling layer processing, encoding features After global average pooling, standard convolution, batch normalization and Sigmoid activation are performed to obtain enhanced features. ;

[0027] Step S44: Residual mechanism processing to enhance features and enhanced features Perform dot product multiplication fusion and combine the fusion result with the encoding feature Add to get enhanced features ;

[0028] Step S5: Multi-layer upsampling to enhance features , optimization features and coded features 、 Perform multi-scale feature fusion based on depthwise separable convolution and three-layer upsampling decoding with three-layer parallel spatial enhancement based on residual mechanism to obtain the segmentation mask:

[0029] ;

[0030] ;

[0031] ;

[0032] ;

[0033] Where, represents the upsampling operation, stands for three-layer parallel spatial enhancement based on residual mechanism, Represents multi-scale feature fusion based on depthwise separable convolution, Represents a standard convolution with a convolution kernel size of 1×1, which is used for dimension alignment before feature connection. 、 、 Represent the results of three-layer upsampling decoding, represents the Sigmoid activation function, represents the segmentation mask;

[0034] Step S6: Segmentation image generation: Generate a lung segmentation map, i.e., a lesion image, based on the segmentation mask. Specifically, the following steps are included:

[0035] Step S61: Load the segmentation mask and the corresponding original image;

[0036] Step S62: converting the segmentation mask into a grayscale image or a binary image to ensure that the value of each pixel represents a different segmentation category or object;

[0037] Step S63: generating a segmentation map according to the segmentation mask, traversing each pixel of the segmentation mask, and determining the pixel value in the segmentation image at the corresponding position according to the pixel value;

[0038] Step S64: Visualize the generated segmented image to display different segmented regions or objects in the image.

[0039] Furthermore, in the cross-modal feature fusion module, a cross-attention multimodal feature fusion method based on bidirectional gating specifically includes the following steps:

[0040] Step Q1: A bimodal feature encoder uses a 3D Swin Transformer to extract local and global features of the lesion image to obtain lesion features. BioClinicalBERT is used to dynamically semantically encode the pathology data text to obtain text features.

[0041] Step Q2: Cross-modal attention, achieving modal interaction through a bidirectional gated attention mechanism:

[0042] Image→text guidance: Lesion features are used to generate a query vector Q, with text features as the key vector K and value vector V. The attention mechanism is used to locate the text semantic focus corresponding to the image features and generate interactive text features.

[0043] Text→Image Guidance: Query vector Q is generated using text features, and lesion features are used as key vector K and value vector V. The attention mechanism is used to enhance the visual saliency of the lesion area and generate interactive visual features.

[0044] Step Q3: Based on the cross-modal feature fusion optimization of channel-spatial attention gating, the interactive text features are optimized based on channel attention to obtain optimized text features, the interactive visual features are optimized based on spatial attention to obtain optimized visual features, and the optimized text features are weightedly fused to obtain cross-modal fusion features.

[0045] The beneficial results achieved by the present invention using the above scheme are as follows:

[0046] (1) This invention creatively adopts a cross-attention multimodal feature fusion method based on bidirectional gating, uses 3D Swin Transformer to extract local and global features of lung CT images, retains the spatial hierarchical structure, uses BioClinicalBERT to dynamically semantically encode pathological text, captures medical entities and time series features in symptom descriptions, and realizes modal interaction through a bidirectional gated attention mechanism, locates the text semantic focus corresponding to image features, enhances the visual significance of the lesion area, introduces a channel-spatial attention gate, and dynamically adjusts the fusion weight of multimodal features, ultimately achieving optimal feature fusion and activation classification, and realizing accurate lung injury detection;

[0047] (2) The present invention creatively adopts a medical image segmentation method based on multi-layer fusion optimization and multi-layer parallel enhancement to segment lung CT images. Through multi-layer feature optimization based on depthwise separable convolution, the detailed local features and global context information of lung CT are effectively extracted. Through three-layer parallel feature enhancement based on the residual mechanism, the key feature, namely the lesion area, is retained. Local-global features are effectively integrated while capturing nonlinear information. Finally, the lung lesion area is accurately segmented on this basis, thereby optimizing cross-modal feature fusion and further increasing the accuracy of lung injury detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 This is a module diagram of the intelligent lung injury detection system based on lung images and pathological data provided by the present invention;

[0049] Figure 2 This is a flowchart of a medical image segmentation method based on multi-layer fusion optimization and multi-layer parallel enhancement;

[0050] Figure 3 The figure is a flowchart of a cross-attention multimodal feature fusion method based on bidirectional gating.

[0051] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention and do not constitute a limitation of the present invention. DETAILED DESCRIPTION

[0052] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments; based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0053] Example 1, see Figure 1, an intelligent lung injury detection system based on lung imaging and pathological data, including a multimodal data acquisition module, an intelligent segmentation module, a cross-modal feature fusion module, and a lung injury classification and detection module;

[0054] The multimodal data acquisition module collects lung CT images from the lung medical imaging database and collects pathological data of patients visiting the pulmonary disease department;

[0055] The intelligent segmentation module uses a medical image segmentation method based on multi-layer fusion optimization and multi-layer parallel enhancement to segment the lung CT image to obtain a lesion image;

[0056] The cross-modal feature fusion module adopts a cross-attention multimodal feature fusion method based on bidirectional gating to perform feature fusion extraction on pathological data and lesion images to obtain cross-modal fusion features;

[0057] The lung injury classification detection module activates and classifies the fused features to obtain a lung injury classification detection result.

[0058] Example 2, see Figure 2 This embodiment is based on the above embodiment. In the intelligent segmentation module, a medical image segmentation method based on multi-layer fusion optimization and multi-layer parallel enhancement specifically includes the following steps:

[0059] Step S1: Multi-layer downsampling: Input the lung CT image into a pre-trained three-layer encoder network for downsampling to obtain three-level encoding features 、 、 ;

[0060] Step S2: Optimize the attention module based on position embedding to encode features Perform position-embedded-based attention module optimization to obtain refined features;

[0061] Step S3: Multi-layer feature optimization based on depth-wise separable convolution to refine features Perform multi-layer feature optimization based on depth-separable convolution to obtain optimized features :

[0062] ;

[0063] Where, stands for batch normalization, represents the ReLU activation function, represents a depthwise separable convolution with a kernel size of k, where k=2i-1, represents the standard convolution with a kernel size of k×k, Representative optimization features ;

[0064] Step S4: Three-layer parallel feature enhancement based on residual mechanism to enhance the encoding features Perform three-layer parallel feature enhancement based on the residual mechanism to obtain enhanced features ;

[0065] Step S5: Multi-layer upsampling to enhance features , optimization features and coded features 、 Perform multi-scale feature fusion based on depthwise separable convolution and three-layer upsampling decoding with three-layer parallel spatial enhancement based on residual mechanism to obtain the segmentation mask:

[0066] ;

[0067] ;

[0068] ;

[0069] ;

[0070] Where, represents the upsampling operation, stands for three-layer parallel spatial enhancement based on residual mechanism, Represents multi-scale feature fusion based on depthwise separable convolution, Represents a standard convolution with a convolution kernel size of 1×1, which is used for dimension alignment before feature connection. 、 、 Represent the results of three-layer upsampling decoding, represents the Sigmoid activation function, represents the segmentation mask;

[0071] Step S6: Segmentation image generation: generating a lung segmentation map, i.e., a lesion image, based on the segmentation mask.

[0072] Embodiment 3: This embodiment is based on the above embodiment, and step S2 specifically includes the following steps:

[0073] Step S21: Feature position encoding embedding, encoding features Perform position encoding to obtain position encoding features;

[0074] Step S22: Matrix mapping, mapping and embedding the position encoding features to obtain the query matrix Q, key matrix K and value matrix V;

[0075] Step S23: self-attention mechanism processing, performing self-attention processing on the query matrix Q, key matrix K and value matrix V to obtain self-attention features;

[0076] Step S24: Feature fusion, encoding features Connect with the self-attention feature to obtain the refined feature .

[0077] Embodiment 4: This embodiment is based on the above embodiment, and step S4 specifically includes the following steps:

[0078] Step S41: Standard convolution and activation layer processing, encoding features Perform standard convolution, batch normalization and ReLU activation to obtain standard features :

[0079] ;

[0080] Where, Representative standard features ;

[0081] Step S42: Based on the local-global feature integration layer processing of the dual-pooling, the standard feature Perform maximum pooling and average pooling respectively, and concatenate the results, perform standard convolution, batch normalization and ReLU activation to obtain enhanced features :

[0082] i 2 =ReLU(BN(Con v 3 [ p 3 avg ( i 1 )⨂ p 3 max ( i 1 )])) ;

[0083] Where, Represents the average pooling process with a convolution kernel size of 3×3, Represents the maximum pooling process with a convolution kernel size of 3×3, Represents a connection operation, Represents enhanced features ;

[0084] Step S43: Global pooling layer processing, encoding features After global average pooling, standard convolution, batch normalization and Sigmoid activation are performed to obtain enhanced features. ;

[0085] Step S44: Residual mechanism processing to enhance features and enhanced features Perform dot product multiplication fusion and combine the fusion result with the encoding feature Add to get enhanced features .

[0086] Example 5, see Figure 3 Based on the above embodiment, this embodiment provides a cross-attention multimodal feature fusion method based on bidirectional gating in the cross-modal feature fusion module, which specifically includes the following steps:

[0087] Step Q1: A bimodal feature encoder uses a 3D Swin Transformer to extract local and global features of the lesion image to obtain lesion features. BioClinicalBERT is used to dynamically semantically encode the pathology data text to obtain text features.

[0088] Step Q2: Cross-modal attention, achieving modal interaction through a bidirectional gated attention mechanism:

[0089] Image→text guidance: Lesion features are used to generate a query vector Q, with text features as the key vector K and value vector V. The attention mechanism is used to locate the text semantic focus corresponding to the image features and generate interactive text features.

[0090] Text→Image Guidance: Query vector Q is generated using text features, and lesion features are used as key vector K and value vector V. The attention mechanism is used to enhance the visual saliency of the lesion area and generate interactive visual features.

[0091] Step Q3: Based on the cross-modal feature fusion optimization of channel-spatial attention gating, the interactive text features are optimized based on channel attention to obtain optimized text features, the interactive visual features are optimized based on spatial attention to obtain optimized visual features, and the optimized text features are weightedly fused to obtain cross-modal fusion features.

[0092] Example 6: This example is based on the above example and the system training design is as follows:

[0093] Loss Function

[0094] Classification loss: Cross entropy loss is used for disease classification;

[0095] Optimizer and learning rate

[0096] Use AdamW optimizer, and the learning rate strategy adopts learning rate decay;

[0097] Training and Inference

[0098] Data augmentation:

[0099] Image: Enhanced using flipping and cropping;

[0100] Text: Use synonyms to replace and cover;

[0101] reasoning:

[0102] Input the patient's CT image and pathology text, and output the lung disease classification results.

[0103] Example 7: This example is based on the above example, and the classification and detection rules for lung injury are as follows:

[0104] pneumonia:

[0105] Including different types such as bacterial pneumonia and viral pneumonia;

[0106] Detection manifestations: CT scans show features such as infiltration shadows;

[0107] Tuberculosis:

[0108] Pulmonary tuberculosis lesions;

[0109] Detection manifestations: the presence of nodules, cavities and other features;

[0110] Lung cancer:

[0111] including malignant and benign tumors;

[0112] Detection manifestations: the presence of abnormal structures such as lung masses and tumors;

[0113] Emphysema:

[0114] Detection manifestations: destruction and overinflation of lung tissue;

[0115] Pulmonary edema:

[0116] Test findings: Imaging changes caused by fluid accumulation in lung tissue;

[0117] Pulmonary embolism:

[0118] Detection manifestations: pulmonary embolism caused by thrombosis in the pulmonary artery or its branches;

[0119] Pulmonary ischemia-reperfusion injury:

[0120] Detection manifestations: There are uneven infiltration shadows, thickening of the alveolar wall, blurred lung texture and lung contours, and blurred pleural contours.

[0121] Example 8: This example is based on the above example and is an optimized implementation of a cross-attention multimodal feature fusion method based on bidirectional gating:

[0122] Step Q1: Bimodal feature encoding

[0123] Image encoding: 3D Swin Transformer is used to extract local and global features of lung CT images, preserving the spatial hierarchical structure;

[0124] Text encoding: Use BioClinicalBERT to dynamically encode pathology text, capturing medical entities and time series features in symptom descriptions;

[0125] Step Q2: Cross-modal attention mechanism optimization

[0126] Modal interaction is achieved through a bidirectional gated attention mechanism:

[0127] Image→Text Guidance: Use CT features to generate a query vector (Q), with text features as key / value (K / V), to locate the text semantic focus corresponding to the image features;

[0128] Text→Image Guidance: Q is generated using text features, and CT features are used as K / V to enhance the visual saliency of the lesion area.

[0129] Step Q3: Multi-scale feature fusion

[0130] A channel-spatial attention gate is introduced to dynamically adjust the fusion weights of multimodal features. A pyramid fusion structure is designed to concatenate and convolute cross-modal features at four scales (original resolution → 1 / 8 downsampling) to capture fine-grained lesion features.

[0131] Dynamic Sparse Attention Mechanism

[0132] Introducing Top-K sparsification during cross-modal interactions, retaining only the top 20% of attention connections;

[0133] Step Q4: Lesion localization-classification joint network

[0134] Adopting double-branch structure:

[0135] Localization branch: Regression of lesion area coordinates (such as ground-glass opacity bounding box) through heatmap

[0136] Classification branch: Combines multimodal features to output disease type probability (pneumonia / lung cancer / fibrosis, etc.).

[0137] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus.

[0138] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

[0139] The present invention and its embodiments are described above. This description is not restrictive. The drawings show only one embodiment of the present invention, and the actual structure is not limited thereto. In short, if a person skilled in the art is inspired by this and, without departing from the purpose of the present invention, designs structures and embodiments similar to this technical solution without inventiveness, they shall fall within the scope of protection of the present invention.

Claims

1. An intelligent lung injury detection system based on lung imaging and pathological data, characterized by: It includes multimodal data acquisition module, intelligent segmentation module, cross-modal feature fusion module and lung injury classification and detection module; The multimodal data acquisition module collects lung CT images from the lung medical imaging database and collects pathological data of patients visiting the pulmonary disease department; The intelligent segmentation module uses a medical image segmentation method based on multi-layer fusion optimization and multi-layer parallel enhancement to segment the lung CT image to obtain a lesion image; The medical image segmentation method based on multi-layer fusion optimization and multi-layer parallel enhancement specifically includes the following steps: Step S1: Perform three-layer downsampling on the lung CT image to obtain three-level coding features 、 、 ; Step S2: Encoding features Perform position-embedded-based attention module optimization to obtain refined features; Step S3: Refine features Perform multi-layer feature optimization based on depth-separable convolution to obtain optimized features : ； Where, stands for batch normalization, represents the ReLU activation function, represents a depthwise separable convolution with a kernel size of k, where k=2i-1, represents the standard convolution with a kernel size of k×k, Representative optimization features ; Step S4: Encoding features Perform three-layer parallel feature enhancement based on the residual mechanism to obtain enhanced features , specifically including the following steps: Step S41: Encoding features Perform standard convolution, batch normalization and ReLU activation to obtain standard features ; Step S42: Standard features Perform maximum pooling and average pooling respectively, and concatenate the results, perform standard convolution, batch normalization and ReLU activation to obtain enhanced features ; Step S43: Encoding features Perform global average pooling, standard convolution, batch normalization and Sigmoid activation to obtain enhanced features ; Step S44: Enhance the features and enhanced features After fusion, with the encoding features Add to get enhanced features ; Step S5: Enhance the features , optimization features and coded features 、 Perform multi-scale feature fusion based on depthwise separable convolution and three-layer upsampling decoding with three-layer parallel spatial enhancement based on residual mechanism to obtain segmentation mask; Step S6: generating a lung segmentation map, i.e., a lesion image, according to the segmentation mask; The cross-modal feature fusion module adopts a cross-attention multimodal feature fusion method based on bidirectional gating to perform feature fusion extraction on pathological data and lesion images to obtain cross-modal fusion features; The lung injury classification detection module activates and classifies the fused features to obtain a lung injury classification detection result.

2. The intelligent lung injury detection system based on lung images and pathological data according to claim 1, characterized in that: In the cross-modal feature fusion module, a cross-attention multimodal feature fusion method based on bidirectional gating specifically includes the following steps: Step Q1: Use the visual feature encoder to extract local and global features of the lesion image to obtain lesion features, and use the text feature encoder to dynamically semantically encode the pathology data text to obtain text features; Step Q2: Implement modal interaction through a bidirectional gated attention mechanism to obtain interactive text features and interactive visual features; Step Q3: Optimize the interactive text features based on channel attention to obtain optimized text features, optimize the interactive visual features based on spatial attention to obtain optimized visual features, and perform weighted fusion on the optimized text features to obtain cross-modal fusion features.

Images