Heterogeneous cross-modal fusion-based pulmonary embolism auxiliary diagnosis method and system

Abstract

The invention relates to the technical field of heterogeneous cross-modal fusion, and discloses a pulmonary embolism auxiliary diagnosis method and system based on heterogeneous cross-modal fusion, and the system comprises the following modules: an image module which combines Swin Transform and an improved 3D CNN model to extract features; the polarization self-attention module is used for efficiently processing high-resolution 3D input features and reducing calculation overhead; the multi-scale feature fusion module integrates SE module channel feature extraction, ASPP module multi-scale feature extraction and 3D convolution space feature extraction; the EHR module is used for extracting features from the electronic health records; and the fusion module is composed of a multi-modal fusion module MAFM and is responsible for extracting key features from the image and the EHR module and fusing the key features so as to improve the diagnosis effect. Through a parallel network and a multi-modal fusion mechanism, the diagnosis precision and robustness of pulmonary embolism are remarkably improved.

Description

Technical Field

[0001] The present invention belongs to but is not limited to the technical field of heterogeneous cross-modal fusion, and in particular relates to a method and system for auxiliary diagnosis of pulmonary embolism based on heterogeneous cross-modal fusion. Background Art

[0002] Pulmonary embolism (PE) is a common cardiopulmonary disease with a high mortality rate, and its incidence rate is increasing year by year. Timely and accurate diagnosis of PE is crucial to reducing mortality. Traditional methods of diagnosing PE mainly rely on CTPA (computed tomography of the pulmonary artery) images, combined with the patient's clinical symptoms and blood biomarkers for comprehensive judgment. Although CTPA has become the "gold standard" for diagnosing PE, since image reading and interpretation require experienced radiologists, there are problems of high subjectivity and time-consuming, and in some cases, misdiagnosis or missed diagnosis may occur.

[0003] In recent years, the development of artificial intelligence (AI) and machine learning technology in the medical field has provided new diagnostic methods, especially in the field of medical image analysis. Automated image processing and analysis technology has made AI-based intelligent auxiliary diagnosis systems gradually become effective clinical tools. Such systems greatly improve the efficiency and accuracy of diagnosis by automatically analyzing lesions in CT images. However, relying solely on images for diagnosis often ignores other important clinical information of patients, such as historical medical conditions, laboratory test data, and vital signs in electronic health records (EHR). With the emergence of multimodal data fusion technology, the combination of image data and structured clinical data has brought new possibilities for the diagnosis of pulmonary embolism.

[0004] Multimodal data fusion technology can simultaneously utilize anatomical information in CT images and medical history information in EHR data to achieve heterogeneous cross-modal integrated analysis, helping to more accurately determine the type, severity, and potential causes of pulmonary embolism. This technological development has brought new directions to the diagnosis of pulmonary embolism, but the existing system still faces challenges in the mechanism of data fusion.

[0005] At present, medical image analysis technology has made some progress in the automatic diagnosis of pulmonary embolism. Deep learning models such as 3D U-Net are widely used in CT image segmentation for automatic detection of PE lesions. Other advanced models, such as Swin Transformer, also show strong performance in feature extraction and image classification. These aspects can detect pulmonary embolism lesions more accurately through training on CT images. However, this type of model generally relies on a single image data, ignoring important information in the patient's clinical data (such as blood oxygen saturation, D-dimer level, previous medical history, etc.), and cannot achieve a comprehensive assessment of the patient's condition.

[0006] To overcome this limitation, a few studies have begun to try to combine structured EHR data with CT images and use cross-modal fusion technology for comprehensive diagnosis of pulmonary embolism. By fusing EHR and CT image data, the diagnostic model can not only identify lesions in the image, but also combine clinical data to provide more personalized diagnostic recommendations. However, the current cross-modal fusion technology still faces some technical bottlenecks:

[0007] Data heterogeneity problem: EHR data and CT data come from different data modalities. The former is structured tabular data, and the latter is unstructured three-dimensional images. How to effectively process and fuse these two types of data is currently a difficult problem. Existing methods often use simple splicing or weighted averaging to fuse these data, but this method cannot fully utilize the potential interaction characteristics between the two, resulting in limited improvement in diagnostic performance.

[0008] Insufficient feature extraction and representation: EHR data contains rich patient history and clinical information, while imaging data provides anatomical and pathological information. Existing technologies are still insufficient in extracting diagnostically useful features from these two types of data, especially when the data size is limited. Models are prone to overfitting and are difficult to effectively generalize to clinical applications.

[0009] Challenges of cross-modal learning: Current neural network architectures used for multimodal data fusion, such as simple multi-layer perceptrons (MLPs) or convolutional neural networks (CNNs), lack sufficient adaptability when processing imaging and EHR data. How to design a more effective model structure that can adaptively capture the association between different modal data is an important direction of current research.

[0010] Although these methods have improved the diagnostic performance to a certain extent, due to data heterogeneity and the limitations of existing model fusion mechanisms, the application of cross-modal fusion technology in intelligent assisted diagnosis of pulmonary embolism is far from mature, and a more effective fusion mechanism is urgently needed to improve diagnostic accuracy. Summary of the invention

[0011] In view of the problems existing in the prior art, the present invention provides a method and system for auxiliary diagnosis of pulmonary embolism based on heterogeneous cross-modal fusion.

[0012] The present invention is implemented as follows: a pulmonary embolism auxiliary diagnosis system based on heterogeneous cross-modal fusion, the system comprising:

[0013] In the imaging module, a feature extraction model combining Swin Transformer and an improved version of 3D CNN is established; the hybrid architecture uses Swin Transformer and CNN in parallel to extract more comprehensive features at different levels; in order to improve the network's ability to capture global and local features, a multi-scale feature fusion MSFF module is introduced; this module fuses the features extracted by Swin Transformer and 3D CNN at different scales; to further enhance the feature extraction capability of 3D CNN, a polarized self-attention PSA module is added; in the overall architecture, multiple layers of residual convolutional layers and convolutional layers are used to ensure that the features from the two branches can be effectively fused and improve the training effect of the network; these convolutional layers help to fuse and downsample features, thereby optimizing the model's performance in feature extraction and integration;

[0014] Polarized self-attention module, so that the model can efficiently process high-resolution 3D input features with minimal computational overhead, while effectively capturing long-range dependencies for accurate voxel-level predictions; introduced polarized self-attention PSA module; this module consists of two components: polarized channel self-attention PCSA submodule and polarized spatial self-attention PSSA submodule;

[0015] Multi-scale feature fusion module, which integrates channel feature extraction from the squeeze and excitation SE module, multi-scale feature extraction from the dilated spatial pyramid pooling ASPP module, and spatial feature extraction from 3D convolution; this approach makes the feature fusion of SwinTransformer and 3DCNN encoder more effective at each level;

[0016] EHR module, in order to effectively extract features from electronic health record EHR data, the six table files were first systematically standardized; this process includes removing features with zero variance and standardizing the remaining features by subtracting the mean and dividing by the standard deviation; this normalization ensures that all features are on a comparable scale, facilitating meaningful comparisons; subsequently, multiple table data sets are merged into a unified table based on the patient index number; feature and attention transformers FATs are proposed; this module integrates a feature transformer, an attention transformer, and a new branch; the feature transformer is responsible for basic feature transformation, using a gated linear unit GLU to extract features and perform nonlinear transformations on them; this design enables the model to effectively capture the complex relationships between features; the attention transformer applies an attention mechanism to weight features to highlight their importance; the new branch consists of a fully connected layer and an adaptive pooling layer; the combination of the fully connected layer and the adaptive pooling enables the branch to effectively aggregate global context while dynamically adjusting the feature space;

[0017] The fusion module includes three different multimodal fusion modules MAFM; during the implementation process, key features are first extracted from the imaging module and the EHR module; then, these features are input into the MAFM for comprehensive fusion to optimize the diagnostic effect and enhance the complementarity between the modalities.

[0018] Furthermore, the polarized channel self-attention PCSA submodule enhances the channel-level feature representation by capturing global context and detailed channel dependencies. First, the PCSA submodule generates Query (Q ch )、Key(K ch ) and Value(V ch ) tensor; these convolutions reduce the computational complexity and number of parameters by decomposing the convolution operation into depth convolution and point convolution while maintaining effective feature representation; the channel attention score matrix (Z ch ) through the query matrix (Q ch ) and the key matrix (K ch ) is calculated by multiplying the score matrix with the value matrix (V ch ) is multiplied element by element to enhance the model's ability to extract specific channel features; assuming that the input feature map is Channel attention score matrix The calculation formula is as follows:

[0019] Q ch =σ q (W q (X)

[0020] K ch =F SM (σ k (W k (X)))

[0021] Z ch =F SG (F LN (W z (Q ch ×K ch )))

[0022] Among them, W q , W k and W z They represent the convolutional layer, σ q and σ k represents the reshaping operation, F SM represents the Softmax function; F LN represents layer normalization, and F SG is the sigmoid activation function;

[0023] The final output of the PCSA module is the enhanced channel feature map

[0024] V ch =W v (X)

[0025] X ch =V ch ⊙Z ch

[0026] Among them, W v is a convolutional layer, and ⊙ represents element-wise multiplication.

[0027] Furthermore, the polarized spatial self-attention PSSA submodule: The PSSA submodule enhances the spatial feature representation by capturing detailed spatial dependencies; the submodule first generates a query vector and a global spatial key matrix (K sp ), and use depth-wise separable convolution to implement it; then, the global spatial key matrix is ​​combined with all query vector matrices (Q sp ) to obtain the spatial attention score matrix (Z sp ). Next, the score matrix and the value matrix (V sp ) are element-wise multiplied to improve the extraction of space-specific features. In the final stage of the PSSA submodule, bottleneck features are extracted from the enhanced features;

[0028] The input of the PSSA module is the channel enhanced feature map Spatial Attention Score Matrix The calculation formula is as follows:

[0029] Q sp =σ q (P q (W q (X ch )))

[0030]

[0031] Z sp =F SG (σ z (K sp ×Q sp ))

[0032] Among them, W q and W k is the convolutional layer, σ q , and σ z Represent the reshaping operation, P q and P k is the dimension transformation operation, F GM represents the global average operation, FSM is the softmax function, F SG is the sigmoid activation function;

[0033] The final output of the PSSA submodule is the spatial enhanced feature map

[0034] V sp =P v (σ v (W v (X ch )))

[0035] X sp =W out (σ out (P out (V sp ⊙Z sp )))

[0036] Among them, W v and W out is the convolutional layer; P v and P out is the dimension transformation operation; σ v and σ out represents a reshape operation; ⊙ represents an element-wise multiplication.

[0037] Furthermore, the multi-scale feature fusion module specifically includes:

[0038] Assume that the intermediate features of the two parallel encoders at the same level are Attention Weight The calculation formula is as follows:

[0039] w=σ(W se (X+Y)+W aspp (X+Y)+W 3dconv (X+Y))

[0040] Among them, W se (X+Y), W aspp (X+Y) and W 3dconv (X+Y) represents the output of SE module, ASPP module and 3D convolution respectively, σ represents Sigmoid activation function;

[0041] Squeeze and Excitation (SE) module: The SE module enhances the feature representation of the channel dimension by adaptively weighting each channel, thereby increasing the focus on important features. The specific calculation is as follows:

[0042] W se (X+Y)=(X+Y)·f se

[0043] f se =σ(W se1 f1)

[0044] f1=ReLU(W se2 (GlobalAvgPool(X+Y)))

[0045] Where W se1 and W se2 is the learned weight matrix, GlobalAvgPool(·) represents the global average pooling operation;

[0046] Expanded Spatial Pyramid Pooling (ASPP) module: The ASPP module uses expanded convolutions of different scales to capture multi-scale contextual information and enhance the model's ability to represent features under different receptive fields. The specific calculation is as follows:

[0047] W aspp (X+Y)=W aspp ·f aspp

[0048] f aspp =Conv 1×1×1 (f aspp1 )

[0049]

[0050] Z=Conv 1×1×1 (GlobalAvgPool(X+Y))

[0051] Among them, r1, r2 and r3 represent the expansion rates set to 6, 12 and 18 respectively;

[0052] 3D convolution module: The 3D convolution module enhances feature extraction in the spatial dimension, helping to capture complex spatial information and contextual relationships. The specific calculation is as follows:

[0053] W 3dconv (X+Y)=Conv 1×1×1 (V)+(X+Y)

[0054] V = GELU(BN(Conv 3×3×3 (X+Y)))

[0055] In order to integrate features from different branches and introduce adaptive weights, a Sigmoid activation function is applied to map the input features to the range of 0 to 1; the calculated attention weights are used to weight the input features, which are then fused to obtain the final output feature map. The specific calculation is as follows:

[0056] X'=X·w

[0057] Y'=Y·w

[0058] Z=Concat(X'+Y')

[0059] In order to ensure that the features of each level are effectively utilized in pulmonary embolism detection, skip connections are used. Each encoder layer contains a residual unit with a 3D convolution layer of a specified size, an adaptive maximum pooling layer, and instance normalization; the features processed at each level are spatially reduced by half and connected with the results of multi-scale feature fusion to facilitate the subsequent pulmonary embolism detection task.

[0060] Furthermore, the EHR module specifically includes:

[0061] (1) Feature transformer

[0062] Let the input be a one-dimensional feature vector Where B is the batch size and D is the feature dimension. The processing flow of the feature transformer is described as follows:

[0063] In the Feature transformer, each feature block consists of a fully connected layer (FC), a batch normalization layer (BN), and a GLU layer. The input features are first linearly transformed by the fully connected layer, then batch normalized, and finally nonlinearly transformed by the GLU layer. This combination provides powerful expressive power for the features. The feature transformer contains multiple GLU layers, of which the first two are shared in all decision steps to ensure the consistency of the feature transformation process. These shared GLU layers maintain the unity of capturing global features by applying the same parameters in different decision steps. The last two GLU layers depend on the decision step, allowing personalized adjustments based on the feature state of each decision point. This design enriches the feature representation and provides flexibility between different decision steps. The outputs of these feature blocks are combined with normalization through residual connections, where the normalization factor is This normalization maintains the stability of feature variance during the learning process, avoiding drastic fluctuations in network training, thereby enhancing the stability of the overall training. This structure mimics the decision tree, which helps to effectively integrate and synthesize input feature information and capture complex feature interactions.

[0064] (2) Attentive transformer

[0065] The Attentive Transformer consists of four main components: a fully connected layer (FC), a batch normalization layer (BN), a priori scaling layer, and a sparse maximization layer. This module enhances the learning ability of the model and improves the efficiency of feature selection through effective feature selection and masking mechanisms.

[0066] In the Attentive transformer, the batch normalized features are element-wise multiplied with the prior scaling factor P[i], which is defined as follows:

[0067]

[0068] Among them, γ is a relaxation parameter, M[j] represents the mask feature from the previous stage;

[0069] The core task of the Attentive Transformer is to determine the mask layer of the current step based on the result of the previous step; important features are learned through masks is sparsely selected; the masking mechanism ensures that the model's learning ability is focused on relevant features, thereby improving parameter efficiency; the attention transformer generates a mask from the processed features through the following formula:

[0070] M[i]=sparsemax(P[i-1]*h i (a[i-1]))

[0071] Among them, h i is a trainable function consisting of a fully connected layer (FC layer) and a batch normalization layer (BN layer), P[i-1] is the previous scaling factor from the previous stage, and sparsemax(·) is the activation function used to generate the sparse mask, defined as:

[0072] sparsemax(x)=max(0,x-τ)

[0073] Where τ is a threshold value, calculated as follows:

[0074]

[0075] here, Represents x j -The positive part of τ, k is the number of features to be retained. In the sparse maximization process, τ is a minimum value to ensure that exactly k features are retained.

[0076] Furthermore, the fusion module specifically includes:

[0077] For the image feature x i and the table features y i , calculate the matching score βj,i and ρ j,i To evaluate the similarity between image features and table features; these scores are calculated by the dot product of the query vector and the key vector, and then normalized using the Softmax function, as follows:

[0078]

[0079] Among them, q1(x i )、k1(x j ),q2(y i ) and k2(y j ) represent the image features x after linear transformation. i and the table features y i The query vector and key vector of ;

[0080] In order to achieve the calculation of the above matching scores, three different fusion modules are designed;

[0081] Softmax FusionModule: Inspired by the bidirectional multimodal attention mechanism, a Softmax fusion module is proposed, which realizes effective feature interaction by utilizing the matching score formula. Specifically, it starts from the input image feature X I and table feature X T The query vector (Q), key vector (K) and value vector (V) are generated in the following way:

[0082] Q I ,K I ,V I =LP(Norm(X I ))

[0083] Q T ,K T ,V T =LP(Norm(X T ))

[0084] Among them, LP(·) represents the linear projection operation and Norm(·) represents layer normalization. The definition of multimodal attention fusion is:

[0085] F I =Attention(Q I ,K I ,V I )+Attention(Q I ,K T ,V T )

[0086] F T =Attention(QT ,K T ,V T )+Attention(Q T ,K I ,V I )

[0087] Here Attention(Q,K,V) consists of two matrix multiplications and a scaled Softmax operation:

[0088]

[0089] Where T represents the matrix transpose, d k is a scaling hyperparameter; the attention results are combined with the original features through residual connections to prevent information loss and stabilize the training process:

[0090] χ I =F I +X I

[0091] χ T =F T +X T

[0092] The fused image features and table features are then input into a fully connected layer to generate the final prediction results of pulmonary embolism;

[0093] NonlinearFusion Module: The nonlinear fusion module introduces the kernel function transformation φ(·) to perform nonlinear transformation on the query vector and the key vector, so that the model can capture more complex feature relationships beyond the linear matching score formula. The specific calculation is as follows:

[0094]

[0095] Where φ(·)=elu(x)+1 represents the kernel function. Inspired by the linear attention mechanism, the definition of multimodal attention fusion is:

[0096]

[0097] This kernel function transformation enhances the model’s ability to capture complex nonlinear relationships, enabling the nonlinear fusion module to better understand the interactions between multimodal data. Through residual connections, the attention results are combined with the original features, and the fused features are input into the fully connected layer to generate the final pulmonary embolism prediction.

[0098] Hybrid Fusion Module: The hybrid fusion module combines features from the Softmax and nonlinear fusion modules; generates query (Q), key (K), and value (V) vectors, and applies kernel function transformation to the query and key vectors of each modality; however, unlike the nonlinear fusion module, the hybrid fusion module retains the multiplication order of the Softmax fusion module, and the multimodal attention fusion formula is:

[0099]

[0100] In F I and F T In the formula, the first term represents the self-attention mechanism in the image modality and the table modality respectively, while the second term represents the cross-modal attention mechanism; by combining the kernel function transformation with the traditional attention mechanism, the hybrid fusion module is able to handle linear and nonlinear relationships, provide flexibility and capture more comprehensive cross-modal interactions; the attention results are integrated with the original features through residual connections, and the fused features are input into the fully connected layer to generate the final prediction of pulmonary embolism.

[0101] Another object of the present invention is to provide a pulmonary embolism auxiliary diagnosis method based on heterogeneous cross-modal fusion based on the pulmonary embolism auxiliary diagnosis system based on heterogeneous cross-modal fusion as described in claims 1-6, the method specifically comprising:

[0102] S1: A feature extraction model combining SwinTransformer and an improved version of 3D CNN is established; the hybrid architecture uses SwinTransformer and CNN in parallel to extract more comprehensive features at different levels; in order to improve the network's ability to capture global and local features, a multi-scale feature fusion MSFF module is introduced; this module fuses the features extracted by SwinTransformer and 3D CNN at different scales; the feature extraction capability of 3D CNN is further enhanced by adding a polarized self-attention PSA module; in the overall architecture, multiple layers of residual convolutional layers and convolutional layers are used to ensure that the features from the two branches can be effectively fused and improve the training effect of the network; these convolutional layers help to fuse and downsample the features, thereby optimizing the model's performance in feature extraction and integration;

[0103] S2: Utilizes the polarized self-attention module so that the model can efficiently process high-resolution 3D input features with minimal computational overhead while effectively capturing long-range dependencies for accurate voxel-level predictions; introduces the polarized self-attention PSA module; this module consists of two components: the polarized channel self-attention PCSA submodule and the polarized spatial self-attention PSSA submodule;

[0104] S3: Using the multi-scale feature fusion module, we integrate the channel feature extraction from the squeeze and excitation SE module, the multi-scale feature extraction from the dilated spatial pyramid pooling ASPP module, and the spatial feature extraction from 3D convolution; this method makes the feature fusion of the Swin Transformer and 3D CNN encoder more effective at each level;

[0105] S4: In order to effectively extract features from electronic health record (EHR) data, the six tabular files were first systematically standardized. This process involved removing features with zero variance and standardizing the remaining features by subtracting the mean and dividing by the standard deviation. This normalization ensured that all features were on comparable scales, facilitating meaningful comparisons. Subsequently, multiple tabular datasets were merged into a unified table based on the patient index number. Feature and attention transformers (FATs) were proposed. This module integrates a feature transformer, an attention transformer, and a new branch. The feature transformer is responsible for basic feature transformation, using a gated linear unit (GLU) to extract features and perform nonlinear transformations on them. This design enables the model to effectively capture complex relationships between features. The attention transformer applies an attention mechanism to weight features to highlight their importance. The new branch consists of a fully connected layer and an adaptive pooling layer. The combination of the fully connected layer and the adaptive pooling enables the branch to effectively aggregate global context while dynamically adjusting the feature space.

[0106] S5: Utilize fusion modules, including three different multimodal fusion modules (MAFM); during the implementation process, key features are first extracted from the imaging module and the EHR module; then, these features are input into the MAFM for comprehensive fusion to optimize the diagnostic effect and enhance the complementarity between the modalities.

[0107] Another object of the present invention is to provide a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the method for auxiliary diagnosis of pulmonary embolism based on heterogeneous cross-modal fusion.

[0108] Another object of the present invention is to provide a computer-readable storage medium storing a computer program, which, when executed by a processor, enables the processor to perform any step of the method for auxiliary diagnosis of pulmonary embolism based on heterogeneous cross-modal fusion.

[0109] Another object of the present invention is to provide an information data processing terminal, which is used to implement the pulmonary embolism auxiliary diagnosis system based on heterogeneous cross-modal fusion.

[0110] In combination with the above technical solutions and the technical problems solved, the advantages and positive effects of the technical solutions to be protected by the present invention are as follows:

[0111] First, 1. Innovative parallel network design

[0112] In the prior art, a single encoder (whether it is a Swin Transformer or a CNN encoder) can usually only extract feature information of a certain dimension from a CT image, resulting in incomplete information extraction when diagnosing complex lesions. In order to solve this problem, the present invention proposes a parallel network structure that combines the Swin Transformer encoder and the CNN encoder to extract global and local feature information from CT images, respectively. Through this design, the network can capture key lesions in CT images more comprehensively and effectively, improving the accuracy and efficiency of pulmonary embolism detection. Especially in complex CT images, this structure can better cope with the details and global information of the image, and optimize the integrity and accuracy of information extraction.

[0113] 2. Multi-scale feature fusion module (MSFF)

[0114] The existing cross-encoder feature fusion technology has the problem of poor fusion effect, and usually cannot give full play to the advantages of features extracted by different encoders. The present invention solves this fusion problem by designing a multi-scale feature fusion module (MSFF). This module can effectively fuse features from different encoders and use multi-scale information processing methods to ensure the complementarity of global and local features. This not only improves the feature expression ability of the pulmonary embolism diagnosis model, but also enhances the robustness of the model in complex cases, thereby greatly improving the accuracy of diagnosis.

[0115] 3. Polarized Self-Attention

[0116] The traditional 3D CNN attention mechanism has the problems of high computational complexity and limited global information capture capability, which becomes a bottleneck in large-scale 3D image analysis. The present invention introduces a polarized self-attention module, which significantly reduces computational complexity and improves the breadth of information capture by optimizing the computational structure and focusing on global features. This module improves the efficiency of image feature extraction while ensuring high accuracy in pulmonary embolism diagnosis, especially showing significant technical advantages in large-scale image analysis.

[0117] 4. Feature and Attentive Transformers (FATs)

[0118] When processing structured data such as electronic health records (EHR), the existing technology often has the problem of insufficient feature extraction, which affects the accuracy of diagnosis. The present invention designs the Feature and Attentive Transformers (FATs) module to perform deep feature extraction specifically for EHR data. This module not only improves the processing capability of structured data, but also provides a more reliable data basis for the diagnosis of pulmonary embolism through an efficient feature extraction mechanism. Compared with traditional methods, the FATs module enables the diagnostic model to more accurately capture key information related to the disease, enhancing the comprehensiveness and reliability of the diagnosis.

[0119] 5. Multimodal Attention Fusion Module

[0120] The effective fusion of cross-modal data has always been a difficulty in the prior art, especially in the information alignment and complementarity between CT images and tabular data (such as EHR). The present invention proposes three innovative multimodal attention fusion modules. By evaluating different fusion mechanisms, the modules with the best performance are selected for practical application. This module can accurately align the complementary information in CT images and EHR data, and realize more efficient cross-modal feature fusion, thereby significantly improving the accuracy and robustness of pulmonary embolism diagnosis. Compared with the existing simple data splicing method, this fusion method based on the attention mechanism effectively solves the coordination problem between heterogeneous data and improves the diagnostic performance of the model.

[0121] Advantages of this technology include:

[0122] Improved diagnostic performance: Through parallel networks and multimodal fusion mechanisms, the diagnostic accuracy and robustness of pulmonary embolism are significantly improved.

[0123] Computational efficiency optimization: The introduced polarized self-attention module reduces the computational burden in 3D image processing while maintaining high accuracy.

[0124] Effective fusion of multimodal data: The multimodal attention fusion module can accurately integrate the complementary information in CT images and EHR data, enhancing the diagnostic comprehensiveness of the model.

[0125] Efficient feature extraction: The feature extraction of EHR data by the FATs module improves the utilization efficiency of structured data and provides a more reliable basis for diagnosis.

[0126] Reliable experimental verification: Through experimental verification, this technology has shown superior application value and promotion potential in the diagnosis of pulmonary embolism.

[0127] Second, as auxiliary evidence of the inventiveness of the claims of the present invention, it is also reflected in the following important aspects:

[0128] (1) The expected benefits and commercial value of the technical solution of the present invention after transformation are:

[0129] The present invention realizes the efficient fusion of CT images and EHR data through innovative parallel network design, multi-scale feature fusion, polarized self-attention mechanism and multimodal fusion module, thereby improving the accuracy and robustness of pulmonary embolism diagnosis. The clinical transformation of this technology can significantly shorten the diagnosis time, reduce the risk of misdiagnosis and missed diagnosis, and reduce the workload of doctors. Its wide application is expected to create huge commercial value in the medical industry, especially in the image analysis-assisted diagnosis market. In addition, the present invention can provide hospitals with intelligent pulmonary embolism detection solutions, improve the efficiency of medical resource utilization, and thus reduce overall medical costs. It is expected that the promotion of this technical solution will bring significant economic benefits to medical institutions and provide patients with higher quality medical services.

[0130] (2) The technical solution of the present invention fills the technical gap in the industry at home and abroad:

[0131] In the prior art, the diagnosis of pulmonary embolism usually relies on a single-modality image analysis system, which cannot fully utilize structured and unstructured data, resulting in low diagnostic accuracy in complex cases. In addition, due to the high computational complexity of existing 3D image analysis technology, it is difficult to efficiently process large-scale data, resulting in a technical gap in the diagnosis of diseases such as pulmonary embolism. The present invention combines a parallel network structure with a polarized self-attention mechanism to improve the accuracy of image information capture and reduce computational costs; the multimodal attention fusion module realizes the complementary fusion of CT images and EHR data, filling the technical gap in efficient alignment and fusion of cross-modal data at home and abroad, and providing a new technical path for intelligent auxiliary diagnosis of complex diseases such as pulmonary embolism.

[0132] (3) The technical solution of the present invention solves the technical problems that people have been eager to solve but have never been able to solve successfully:

[0133] The clinical diagnosis of pulmonary embolism has long faced the problems of difficulty in accurately extracting image features and difficulty in aligning and fusing multimodal data. Especially in complex cases, existing methods often find it difficult to take into account both global information and detailed features. The present invention successfully solves this problem through an innovatively designed parallel network architecture and multimodal fusion mechanism. Specifically, the SwinTransformer and CNN parallel encoders can comprehensively extract global and local features from CT images, while the multimodal attention fusion module achieves effective complementarity between CT images and EHR data, significantly improving the accuracy and stability of diagnosis. These innovative designs have overcome long-standing technical bottlenecks, making the diagnosis of complex cases possible and greatly promoting the development of intelligent auxiliary diagnosis technology.

[0134] (4) The technical solution of the present invention overcomes technical prejudice:.

[0135] In traditional imaging diagnosis, it is often dependent on a single-modality deep learning network, which leads to deviations in the use of imaging and clinical data, and ignores the synergistic effect of cross-modal data in diagnosis. For the use of structured data such as EHR, traditional methods are often limited by insufficient feature extraction, and it is believed that it has limited value in auxiliary diagnosis of pulmonary embolism, and cross-modal fusion is difficult. The present invention breaks through this technical bias, and deeply mines the feature information of EHR data by designing a dedicated Feature and Attentive Transformers module (FATs), and combines it with a multimodal attention fusion module to effectively integrate structured data into the diagnostic model, making collaborative analysis of CT images and EHR data possible, and improving the comprehensiveness and reliability of diagnosis.

[0136] Third, the existing technical problems solved by the technical solution of the present invention in industrial applications are:

[0137] 1. Problems with existing technologies

[0138] 1. Insufficient feature extraction and fusion of multimodal data

[0139] Existing pulmonary embolism diagnostic systems mostly rely on single-modality data (such as CT images) and are unable to fully utilize non-imaging data such as patients' electronic health records (EHRs), resulting in incomplete diagnostic information and affecting diagnostic accuracy and reliability.

[0140] The feature extraction efficiency of single modality data is low, especially for the identification of complex lesions. It is difficult to capture deep structural information, resulting in inaccurate lesion positioning.

[0141] 2. Limitations of image feature extraction models

[0142] Conventional convolutional neural network (CNN) models have high computational complexity when processing 3D images, which easily leads to unbalanced feature extraction and fails to fully capture multi-scale features.

[0143] The feature extraction process lacks the effective combination of global information and local features, especially in high-resolution 3D data, where the feature integration capability is limited.

[0144] 3. Insufficient interpretability of diagnostic results and decision support

[0145] The "black box" nature of deep learning models leads to a lack of explainability in diagnostic results, making it difficult for doctors to directly understand the model's predictive basis, which affects actual clinical applications.

[0146] Current technologies fail to effectively integrate the diagnostic results of multimodal features into easy-to-understand reports, limiting the practical utility of decision support.

[0147] 4. Insufficient real-time and scalability

[0148] The existing system has a large computational overhead and is unable to meet the rapid response requirements in real-time applications.

[0149] The system lacks modular design and cannot be flexibly expanded or upgraded for specific scenarios.

[0150] Significant technical advancements of the present invention:

[0151] 1. Progress in multimodal data fusion

[0152] The present invention realizes the efficient integration of image and non-image data by extracting and fusing the features of the image module and the EHR module, and utilizing the multi-scale feature fusion module (MSFF) and the multi-modal fusion module (MAFM).

[0153] The introduction of the polarized self-attention module (PSA) effectively captures the long-distance dependencies and key features in multimodal features, providing support for more accurate diagnosis.

[0154] 2. Innovation in image feature extraction

[0155] A hybrid architecture combining Swin Transformer and an improved version of 3D CNN is proposed, which significantly improves the efficiency and accuracy of feature extraction of 3D images.

[0156] Swin Transformer excels in capturing global information, effectively complementing the shortcomings of CNN in extracting local features.

[0157] 3D CNN provides stronger spatial feature processing capabilities, especially in the layered processing of high-resolution data.

[0158] The multi-scale feature fusion module efficiently integrates channel features (SE module), multi-scale context (ASPP module) and spatial features (3D convolution), significantly improving the accuracy of lesion recognition.

[0159] 3. Breakthrough in the explainability of diagnostic results

[0160] The polarized channel self-attention (PCSA) and polarized spatial self-attention (PSSA) modules are introduced to enhance the accuracy of feature extraction while reducing the computational overhead.

[0161] The PCSA module provides a clear channel-level feature representation, helping doctors intuitively understand which features have a greater impact on the prediction results.

[0162] The PSSA module captures fine-scale spatial dependencies, making the interpretation of diagnostic results more concrete.

[0163] The system generates a comprehensive diagnostic report through a fusion module, including lesion localization, risk score and recommended treatment path, which greatly improves the interpretability and clinical applicability of the results.

[0164] 4. Optimization of real-time performance and scalability

[0165] It adopts a modular design architecture, and each module can be optimized and updated independently, supporting flexible expansion to meet the needs of different scenarios.

[0166] The system's efficient feature extraction and fusion algorithms significantly reduce computational complexity, shortening diagnosis time by approximately 40%, thus meeting the needs of real-time diagnosis.

[0167] 5. Clinical application effect

[0168] The system has demonstrated high efficiency and reliability in actual tests:

[0169] The diagnostic accuracy rate has been increased to over 95%, with significant improvements especially in cases of pulmonary embolism with complex lesions and blurred borders.

[0170] The average diagnosis time has been shortened to a few minutes, significantly improving the efficiency of hospital imaging diagnosis.

[0171] Multimodal fusion makes the diagnostic results more comprehensive and provides doctors with more valuable auxiliary decision-making information.

[0172] The present invention not only solves the problem of multimodal feature extraction and fusion in the prior art at the technical level, but also significantly improves the efficiency and accuracy of auxiliary diagnosis of pulmonary embolism by innovatively optimizing the model architecture and algorithm. In industrial applications, the present invention can be widely used in hospital imaging diagnosis centers, telemedicine platforms and medical research institutions, providing strong technical support for the intelligent diagnosis and analysis of complex diseases and promoting the in-depth development of intelligent medicine. BRIEF DESCRIPTION OF THE DRAWINGS

[0173] Figure 1 It is an overall network architecture diagram provided by an embodiment of the present invention;

[0174] Figure 2 is an image module diagram provided by an embodiment of the present invention;

[0175] Figure 3 is a polarization self-attention module provided by an embodiment of the present invention;

[0176] Figure 4 It is a multi-scale feature fusion module provided by an embodiment of the present invention;

[0177] Figure 5 It is Feature and Attentive Transformers provided by the embodiment of the present invention;

[0178] Figure 6 is a Softmax Fusion Module provided by an embodiment of the present invention;

[0179] Figure 7 is a NonlinearFusionModule provided by an embodiment of the present invention;

[0180] Figure 8 A HybridFusion Module provided by an embodiment of the present invention;

[0181] Fig. 9 is a flow chart of a method for auxiliary diagnosis of pulmonary embolism based on heterogeneous cross-modal fusion provided by an embodiment of the present invention;

[0182] Fig.10 Examples of the results of our proposed model, where P represents the confidence score of the predictions, are provided in an embodiment of the present invention;

[0183] Fig.11 AUROC Curves for Ablation Study on MultimodalFusionModules provided by an embodiment of the present invention;

[0184] Fig.12 Confusion Matrices for Ablation Study on Multimodal Fusion Modules provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0185] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0186] like Figure 1 As shown, an embodiment of the present invention provides a pulmonary embolism auxiliary diagnosis system based on heterogeneous cross-modal fusion, the system comprising:

[0187] like Figure 2As shown in the figure, the image module establishes a feature extraction model that combines Swin Transformer and an improved version of 3D CNN; the hybrid architecture uses Swin Transformer and CNN in parallel to extract more comprehensive features at different levels; in order to improve the network's ability to capture global and local features, a multi-scale feature fusion MSFF module is introduced; this module fuses the features extracted by Swin Transformer and 3D CNN at different scales; to further enhance the feature extraction capability of 3D CNN, a polarized self-attention PSA module is added; in the overall architecture, multiple layers of residual convolution layers and convolution layers are used to ensure that the features from the two branches can be effectively fused and improve the training effect of the network; these convolution layers help to fuse and downsample features, thereby optimizing the model's performance in feature extraction and integration;

[0188] Polarized self-attention module, so that the model can efficiently process high-resolution 3D input features with minimal computational overhead, while effectively capturing long-range dependencies for accurate voxel-level predictions; introduced polarized self-attention PSA module; this module consists of two components: polarized channel self-attention PCSA submodule and polarized spatial self-attention PSSA submodule;

[0189] Multi-scale feature fusion module, which integrates channel feature extraction from the squeeze and excitation SE module, multi-scale feature extraction from the dilated spatial pyramid pooling ASPP module, and spatial feature extraction from 3D convolution; this approach makes the feature fusion of the Swin Transformer and 3D CNN encoder more effective at each level;

[0190] EHR module, in order to effectively extract features from electronic health record EHR data, the six table files were first systematically standardized; this process includes removing features with zero variance and standardizing the remaining features by subtracting the mean and dividing by the standard deviation; this normalization ensures that all features are on a comparable scale, facilitating meaningful comparisons; subsequently, multiple table data sets are merged into a unified table based on the patient index number; feature and attention transformers FATs are proposed; this module integrates a feature transformer, an attention transformer, and a new branch; the feature transformer is responsible for basic feature transformation, using a gated linear unit GLU to extract features and perform nonlinear transformations on them; this design enables the model to effectively capture the complex relationships between features; the attention transformer applies an attention mechanism to weight features to highlight their importance; the new branch consists of a fully connected layer and an adaptive pooling layer; the combination of the fully connected layer and the adaptive pooling enables the branch to effectively aggregate global context while dynamically adjusting the feature space;

[0191] The fusion module includes three different multimodal fusion modules MAFM; during the implementation process, key features are first extracted from the imaging module and the EHR module; then, these features are input into the MAFM for comprehensive fusion to optimize the diagnostic effect and enhance the complementarity between the modalities.

[0192] The polarized channel self-attention PCSA submodule enhances the channel-level feature representation by capturing global context and detailed channel dependencies. First, the PCSA submodule generates Query (Q ch )、Key(K ch ) and Value(V ch ) tensor; these convolutions reduce the computational complexity and number of parameters by decomposing the convolution operation into depth convolution and point convolution while maintaining effective feature representation; the channel attention score matrix (Z ch ) through the query matrix (Q ch ) and the key matrix (K ch ) is calculated by multiplying the score matrix with the value matrix (V ch ) is multiplied element by element to enhance the model's ability to extract specific channel features; assuming that the input feature map is Channel attention score matrix The calculation formula is as follows:

[0193] Q ch =σ q (W q (X)

[0194] K ch =F SM (σ k (W k (X)))

[0195] Z ch =F SG (F LN (W z (Q ch ×K ch )))

[0196] Among them, W q , W k and W z They represent the convolutional layer, σ q and σ k represents the reshaping operation, F SM represents the Softmax function; F LN represents layer normalization, and F SG is the sigmoid activation function;

[0197] The final output of the PCSA module is the enhanced channel feature map

[0198] V ch =W v (X)

[0199] X ch =V ch ⊙Z ch

[0200] Among them, W v is a convolutional layer, and ⊙ represents element-wise multiplication.

[0201] like Figure 3 As shown, the polarized spatial self-attention PSSA submodule: The PSSA submodule enhances the spatial feature representation by capturing detailed spatial dependencies; the submodule first generates a query vector and a global spatial key matrix (K) for each attention point. sp ), and use depth-wise separable convolution to implement it; then, the global spatial key matrix is ​​combined with all query vector matrices (Q sp ) to obtain the spatial attention score matrix (Z sp ). Next, the score matrix and the value matrix (V sp ) are element-wise multiplied to improve the extraction of space-specific features. In the final stage of the PSSA submodule, bottleneck features are extracted from the enhanced features;

[0202] The input of the PSSA module is the channel enhanced feature map Spatial Attention Score Matrix The calculation formula is as follows:

[0203] Q sp =σ q (P q (W q (X ch )))

[0204]

[0205] Z sp =F SG (σ z (K sp ×Q sp ))

[0206] Among them, W q and W k is the convolutional layer, σ q , and σ z Represent the reshaping operation, P q and P k is the dimension transformation operation, FGM represents the global average operation, F SM is the softmax function, F SG is the sigmoid activation function;

[0207] The final output of the PSSA submodule is the spatial enhanced feature map

[0208] V sp =P v (σ v (W v (X ch )))

[0209] X sp =W out (σ out (P out (V sp ⊙Z sp )))

[0210] Among them, W v and W out is the convolutional layer; P v and P out is the dimension transformation operation; σ v and σ out represents a reshape operation; ⊙ represents an element-wise multiplication.

[0211] like Figure 4 As shown, the multi-scale feature fusion module specifically includes:

[0212] Assume that the intermediate features of the two parallel encoders at the same level are Attention Weight The calculation formula is as follows:

[0213] w=σ(W se (X+Y)+W aspp (X+Y)+W 3dconv (X+Y))

[0214] Among them, W se (X+Y), W aspp (X+Y) and W 3dconv (X+Y) represents the output of SE module, ASPP module and 3D convolution respectively, σ represents Sigmoid activation function;

[0215] Squeeze and Excitation (SE) module: The SE module enhances the feature representation of the channel dimension by adaptively weighting each channel, thereby increasing the focus on important features. The specific calculation is as follows:

[0216] W se (X+Y)=(X+Y)·fse

[0217] f se =σ(W se1 f1)

[0218] f1=ReLU(W se2 (GlobalAvgPool(X+Y)))

[0219] Where W se1 and W se2 is the learned weight matrix, GlobalAvgPool(·) represents the global average pooling operation;

[0220] Expanded Spatial Pyramid Pooling (ASPP) module: The ASPP module uses expanded convolutions of different scales to capture multi-scale contextual information and enhance the model's ability to represent features under different receptive fields. The specific calculation is as follows:

[0221] W aspp (X+Y)=W aspp ·f aspp

[0222] f aspp =Conv 1×1×1 (f aspp1 )

[0223]

[0224] Z=Conv 1×1×1 (GlobalAvgPool(X+Y))

[0225] Among them, r1, r2 and r3 represent the expansion rates set to 6, 12 and 18 respectively;

[0226] 3D convolution module: The 3D convolution module enhances feature extraction in the spatial dimension, helping to capture complex spatial information and contextual relationships. The specific calculation is as follows:

[0227] W 3dconv (X+Y)=Conv 1×1×1 (V)+(X+Y)

[0228] V = GELU(BN(Conv 3×3×3 (X+Y)))

[0229] In order to integrate features from different branches and introduce adaptive weights, a Sigmoid activation function is applied to map the input features to the range of 0 to 1; the calculated attention weights are used to weight the input features, which are then fused to obtain the final output feature map. The specific calculation is as follows:

[0230] X'=X·w

[0231] Y'=Y·w

[0232] Z=Concat(X'+Y')

[0233] In order to ensure that the features of each level are effectively utilized in pulmonary embolism detection, skip connections are used. Each encoder layer contains a residual unit with a 3D convolution layer of a specified size, an adaptive maximum pooling layer, and instance normalization; the features processed at each level are spatially reduced by half and connected with the results of multi-scale feature fusion to facilitate the subsequent pulmonary embolism detection task.

[0234] Furthermore, the EHR module specifically includes:

[0235] (1) Feature transformer, such as Figure 5 Shown

[0236] Let the input be a one-dimensional feature vector Where B is the batch size and D is the feature dimension. The processing flow of the feature transformer is described as follows:

[0237] In the Feature transformer, each feature block consists of a fully connected layer (FC), a batch normalization layer (BN), and a GLU layer. The input features are first linearly transformed by the fully connected layer, then batch normalized, and finally nonlinearly transformed by the GLU layer. This combination provides powerful expressive power for the features. The feature transformer contains multiple GLU layers, of which the first two are shared in all decision steps to ensure the consistency of the feature transformation process. These shared GLU layers maintain the unity of capturing global features by applying the same parameters in different decision steps. The last two GLU layers depend on the decision step, allowing personalized adjustments based on the feature state of each decision point. This design enriches the feature representation and provides flexibility between different decision steps. The outputs of these feature blocks are combined with normalization through residual connections, where the normalization factor is This normalization maintains the stability of feature variance during the learning process, avoiding drastic fluctuations in network training, thereby enhancing the stability of the overall training. This structure mimics the decision tree, which helps to effectively integrate and synthesize input feature information and capture complex feature interactions.

[0238] (2) Attentive transformer

[0239] The Attentive Transformer consists of four main components: a fully connected layer (FC), a batch normalization layer (BN), a priori scaling layer, and a sparse maximization layer. This module enhances the learning ability of the model and improves the efficiency of feature selection through effective feature selection and masking mechanisms.

[0240] In the Attentive transformer, the batch normalized features are element-wise multiplied with the prior scaling factor P[i], which is defined as follows:

[0241]

[0242] Among them, γ is a relaxation parameter, M[j] represents the mask feature from the previous stage;

[0243] The core task of the Attentive Transformer is to determine the mask layer of the current step based on the result of the previous step; important features are learned through masks is sparsely selected; the masking mechanism ensures that the model's learning ability is focused on relevant features, thereby improving parameter efficiency; the attention transformer generates a mask from the processed features through the following formula:

[0244] M[i]=sparsemax(P[i-1]*h i (a[i-1]))

[0245] Among them, h i is a trainable function consisting of a fully connected layer (FC layer) and a batch normalization layer (BN layer), P[i-1] is the previous scaling factor from the previous stage, and sparsemax(·) is the activation function used to generate the sparse mask, defined as:

[0246] sparse max(x)=max(0,x-τ)

[0247] Where τ is a threshold value, calculated as follows:

[0248]

[0249] here, Represents x j -The positive part of τ, k is the number of features to be retained. In the sparse maximization process, τ is a minimum value to ensure that exactly k features are retained.

[0250] Furthermore, the fusion module specifically includes:

[0251] For the image feature x i and the table features y i , calculate the matching score βj,i and ρ j,i To evaluate the similarity between image features and table features; these scores are calculated by the dot product of the query vector and the key vector, and then normalized using the Softmax function, as follows:

[0252]

[0253] Among them, q1(x i )、k1(x j ),q2(y i ) and k2(y j ) represent the image features x after linear transformation. i and the table features y i The query vector and key vector of ;

[0254] In order to achieve the calculation of the above matching scores, three different fusion modules are designed;

[0255] like Figure 6 As shown in Figure 2, Softmax Fusion Module: Inspired by the bidirectional multimodal attention mechanism, a Softmax fusion module is proposed, which realizes effective feature interaction by utilizing the matching score formula. Specifically, it starts from the input image feature X I and table feature X T The query vector (Q), key vector (K) and value vector (V) are generated in the following way:

[0256] Q I ,K I ,V I =LP(Norm(X I ))

[0257] Q T ,K T ,V T =LP(Norm(X T ))

[0258] Among them, LP(·) represents the linear projection operation and Norm(·) represents layer normalization. The definition of multimodal attention fusion is:

[0259] F I =Attention(Q I ,K I ,V I )+Attention(Q I ,K T ,V T )

[0260] F T=Attention(Q T ,K T ,V T )+Attention(Q T ,K I ,V I )

[0261] Here Attention(Q,K,V) consists of two matrix multiplications and a scaled Softmax operation:

[0262]

[0263] Where T represents the matrix transpose, d k is a scaling hyperparameter; the attention results are combined with the original features through residual connections to prevent information loss and stabilize the training process:

[0264] χ I =F I +X I

[0265] χ T =F T +X T

[0266] The fused image features and table features are then input into a fully connected layer to generate the final prediction results of pulmonary embolism;

[0267] like Figure 7 As shown in the figure, the NonlinearFusionModule: nonlinear fusion module introduces the kernel function transformation φ(·) to perform nonlinear transformation on the query vector and the key vector, so that the model can capture more complex feature relationships beyond the linear matching score formula. The specific calculation is as follows:

[0268]

[0269] Where φ(·)=elu(x)+1 represents the kernel function. Inspired by the linear attention mechanism, the definition of multimodal attention fusion is:

[0270]

[0271] This kernel function transformation enhances the model’s ability to capture complex nonlinear relationships, enabling the nonlinear fusion module to better understand the interactions between multimodal data. Through residual connections, the attention results are combined with the original features, and the fused features are input into the fully connected layer to generate the final pulmonary embolism prediction.

[0272] like Figure 8As shown in the figure, the HybridFusion Module: The hybrid fusion module combines the features from the Softmax and nonlinear fusion modules; generates query (Q), key (K), and value (V) vectors, and applies kernel function transformation to the query and key vectors of each modality; however, unlike the nonlinear fusion module, the hybrid fusion module retains the multiplication order of the Softmax fusion module, and the multimodal attention fusion formula is:

[0273]

[0274] In F I and F T In the formula, the first term represents the self-attention mechanism in the image modality and the table modality respectively, while the second term represents the cross-modal attention mechanism; by combining the kernel function transformation with the traditional attention mechanism, the hybrid fusion module is able to handle linear and nonlinear relationships, provide flexibility and capture more comprehensive cross-modal interactions; the attention results are integrated with the original features through residual connections, and the fused features are input into the fully connected layer to generate the final prediction of pulmonary embolism.

[0275] like Fig. 9 As shown, an embodiment of the present invention provides a pulmonary embolism auxiliary diagnosis method based on heterogeneous cross-modal fusion based on the pulmonary embolism auxiliary diagnosis system based on heterogeneous cross-modal fusion as described in claims 1-6, and the method specifically includes:

[0276] S1: A feature extraction model combining Swin Transformer and improved 3D CNN is established; the hybrid architecture uses Swin Transformer and CNN in parallel to extract more comprehensive features at different levels; in order to improve the network's ability to capture global and local features, a multi-scale feature fusion MSFF module is introduced; this module fuses the features extracted by Swin Transformer and 3D CNN at different scales; further enhances the feature extraction capability of 3D CNN by adding a polarized self-attention PSA module; in the overall architecture, multiple layers of residual convolutional layers and convolutional layers are used to ensure that the features from the two branches can be effectively fused and improve the training effect of the network; these convolutional layers help to fuse and downsample the features, thereby optimizing the model's performance in feature extraction and integration;

[0277] S2: Utilizes the polarized self-attention module so that the model can efficiently process high-resolution 3D input features with minimal computational overhead while effectively capturing long-range dependencies for accurate voxel-level predictions; introduces the polarized self-attention PSA module; this module consists of two components: the polarized channel self-attention PCSA submodule and the polarized spatial self-attention PSSA submodule;

[0278] S3: Using the multi-scale feature fusion module, we integrate the channel feature extraction from the squeeze and excitation SE module, the multi-scale feature extraction from the dilated spatial pyramid pooling ASPP module, and the spatial feature extraction from 3D convolution; this method makes the feature fusion of the Swin Transformer and 3D CNN encoder more effective at each level;

[0279] S4: In order to effectively extract features from electronic health record (EHR) data, the six tabular files were first systematically standardized. This process involved removing features with zero variance and standardizing the remaining features by subtracting the mean and dividing by the standard deviation. This normalization ensured that all features were on comparable scales, facilitating meaningful comparisons. Subsequently, multiple tabular datasets were merged into a unified table based on the patient index number. Feature and attention transformers (FATs) were proposed. This module integrates a feature transformer, an attention transformer, and a new branch. The feature transformer is responsible for basic feature transformation, using a gated linear unit (GLU) to extract features and perform nonlinear transformations on them. This design enables the model to effectively capture complex relationships between features. The attention transformer applies an attention mechanism to weight features to highlight their importance. The new branch consists of a fully connected layer and an adaptive pooling layer. The combination of the fully connected layer and the adaptive pooling enables the branch to effectively aggregate global context while dynamically adjusting the feature space.

[0280] S5: Utilize fusion modules, including three different multimodal fusion modules (MAFM); during the implementation process, key features are first extracted from the imaging module and the EHR module; then, these features are input into the MAFM for comprehensive fusion to optimize the diagnostic effect and enhance the complementarity between the modalities.

[0281] The present invention is an embodiment that provides a computer device, which includes a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the method for auxiliary diagnosis of pulmonary embolism based on heterogeneous cross-modal fusion.

[0282] An embodiment of the present invention provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the processor executes any one of the steps of the method for auxiliary diagnosis of pulmonary embolism based on heterogeneous cross-modal fusion.

[0283] An embodiment of the present invention provides an information data processing terminal, which is used to implement the pulmonary embolism auxiliary diagnosis system based on heterogeneous cross-modal fusion.

[0284] Relevant evidence of the technical effects achieved by the embodiments of the present invention.

[0285] In this study, the RadFusion dataset was used, which contains CT scans and electronic health record (EHR) data from patients with and without pulmonary embolism (PE). The dataset consists of 1,837 axial CTPA examinations, each scanned at an interval of 1.25 mm. Of these, 1,111 scans were negative for PE and 726 were positive for PE. The EHR data provides extensive clinical information, including patient demographics, vital signs (such as systolic blood pressure, diastolic blood pressure, and temperature), 641 medications (including inpatient and outpatient medications), 141 diagnostic groups, and approximately 22 categories of laboratory test results. The EHR data are represented by binary indicators (indicating the presence or absence of the disease) and the most recent test value. The data is structured and contains categorical and numerical information, and no text data is included. In the RadFusion dataset, PE cases are divided into three categories: central PE, segmental PE, and subsegmental PE. There are 257 cases of central PE, 387 cases of segmental PE, and 82 cases of subsegmental PE. To ensure the fairness of the evaluation, the dataset was divided into training set (80%), validation set (10%) and test set (10%) following the standard data splitting strategy of RadFusion. All CT scan images were normalized to the range [-1, 1], while EHR data were normalized to the range [0, 1].

[0286] The model was implemented using PyTorch and MONAI frameworks and trained on an NVIDIA RTX 4090 GPU with 24GB RAM. The SGD optimizer with a learning rate of 0.01 was used and trained for 100 epochs with a batch size of 4. To ensure the stability of the training, a cosine annealing learning rate scheduler was used. The scheduler consisted of 10,000 linear warm-up iterations followed by 600,000 cosine annealing iterations. In terms of data preprocessing, each CT slice was resized to a specific pixel size of 224×224 to improve computational efficiency and randomly cropped to a specified pixel size of 192×192 to enhance data augmentation. In addition, the 3D scanned images were further enhanced by random transformations, including rotation angles to a certain range and jittering up to 8 slices in the depth dimension. Finally, the model was used for training, validation, and test data, achieving the best experimental results.

[0287] 1.1 Comparative Experiment

[0288] Table 1PerformanceComparisonofImaging-Only,EHR-Only,andMultimodalFusionModelswith

[0289] State-of-the-ArtTechniques.

[0290]

[0291] Note: Bold underline denotes the best results in different metrics.

[0292] 2.2. Ablation Experiments

[0293] Table 3 Ablation Study on Image Module.

[0294]

[0295] Note: Bold underline denotes the best results in different metrics.

[0296] Table 4 Ablation Study on EHR Module.

[0297]

[0298] Note: Bold underline denotes the best results in different metrics.

[0299] Table 5 Ablation Study on Multimodal Fusion Modules.

[0300]

[0301] Note: Bold underline denotes the best results in different metrics, bold denotes the second - best results in different metrics.

[0302] It should be noted that the embodiments of the present invention can be implemented by hardware, software, or a combination of software and hardware. The hardware part can be implemented using dedicated logic; the software part can be stored in a memory and executed by an appropriate instruction execution system, such as a microprocessor or dedicated design hardware. It can be understood by a person of ordinary skill in the art that the above-mentioned devices and methods can be implemented using computer executable instructions and / or contained in a processor control code, such as a carrier medium such as a disk, CD or DVD-ROM, a programmable memory such as a read-only memory (firmware), or a data carrier such as an optical or electronic signal carrier. Such code is provided on the carrier medium. The device and its modules of the present invention can be implemented by hardware circuits such as very large-scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, etc., or programmable hardware devices such as field programmable gate arrays, programmable logic devices, etc., can also be implemented by software executed by various types of processors, and can also be implemented by a combination of the above-mentioned hardware circuits and software, such as firmware.

[0303] The above description is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any modifications, equivalent substitutions and improvements made by any technician familiar with the technical field within the technical scope disclosed by the present invention and within the spirit and principle of the present invention should be covered by the protection scope of the present invention.

Claims

1. A pulmonary embolism auxiliary diagnosis system based on heterogeneous cross-modal fusion, characterized in that: The system includes: The imaging module is used to build a feature extraction model that combines Swin Transformer and an improved version of 3D CNN. This module uses a hybrid architecture to use Swin Transformer and CNN in parallel to extract comprehensive features at different levels, and fuses the features extracted by Swin Transformer and 3D CNN at different scales through a multi-scale feature fusion module (MSFF). This module also adds a polarized self-attention module (PSA) to use multiple layers of residual convolutional layers and convolutional layers to achieve effective fusion and downsampling of features, thereby improving feature extraction and integration performance. The polarized self-attention module, which consists of the polarized channel self-attention (PCSA) submodule and the polarized spatial self-attention (PSSA) submodule, is used to efficiently process high-resolution 3D input features, minimize computational overhead, and achieve accurate voxel-level prediction by capturing long-range dependencies; The multi-scale feature fusion module integrates the channel feature extraction of the squeeze and excitation (SE) module, the multi-scale feature extraction of the atrous spatial pyramid pooling (ASPP) module, and the spatial feature extraction of 3D convolution, and realizes the feature fusion of SwinTransformer and 3D CNN encoder at each level; The EHR module is used to extract features from electronic health record (EHR) data, remove features with zero variance by standardization, and standardize the remaining features by mean and standard deviation; merge tabular datasets based on patient index numbers, integrate feature transformers, attention transformers, and new branches through feature and attention transformers (FATs), perform feature transformation using gated linear units (GLUs), and aggregate global context and dynamically adjust feature space through adaptive pooling and fully connected layers; The fusion module, including three multimodal fusion modules (MAFM), is used to receive features from the imaging module and the EHR module for fusion to optimize the diagnosis effect and enhance the complementarity between the modalities.

2. The pulmonary embolism auxiliary diagnosis system based on heterogeneous cross-modal fusion according to claim 1 is characterized in that: The polarized channel self-attention PCSA submodule enhances channel-level feature representation by capturing global context and detailed channel dependencies; First, the PCSA submodule uses depthwise separable convolution to generate Query (Q ch )、Key(K ch ) and Value(V ch ) tensor; these convolutions reduce the computational complexity and number of parameters by decomposing the convolution operation into depth convolution and point convolution while maintaining effective feature representation; the channel attention score matrix (Z ch ) through the query matrix (Q ch ) and the key matrix (K ch ) is calculated by multiplying the score matrix with the value matrix (V ch ) is multiplied element by element to enhance the model's ability to extract specific channel features; assuming that the input feature map is Channel attention score matrix The calculation formula is as follows: Q ch =σ q (W q (X)) K ch =F SM (s k (W k (X))) Z ch =F SG (F LN (W z (Q ch ×K ch ))) Among them, W q , W k and W z They represent the convolutional layer, σ q and σ k represents the reshaping operation, F SM represents the Softmax function; F LN represents layer normalization, and F SG is the sigmoid activation function; The final output of the PCSA module is the enhanced channel feature map V ch =W v (X) X ch =V ch ⊙Z ch Among them, W v is a convolutional layer, and ⊙ represents element-wise multiplication.

3. The pulmonary embolism auxiliary diagnosis system based on heterogeneous cross-modal fusion as claimed in claim 1, characterized in that: The polarized spatial self-attention PSSA submodule: The PSSA submodule enhances the spatial feature representation by capturing detailed spatial dependencies; this submodule first generates a query vector and a global spatial key matrix (K) for each attention point. sp ), and use depth-wise separable convolution to implement it; then, the global spatial key matrix is ​​combined with all query vector matrices (Q sp ) to obtain the spatial attention score matrix (Z sp ); Next, the score matrix and the value matrix (V sp ) element-wise multiplication to improve the extraction of spatial specific features; In the final stage of the PSSA submodule, bottleneck features are extracted from the enhanced features; The input of the PSSA module is the channel enhanced feature map Spatial Attention Score Matrix The calculation formula is as follows: Q sp =σ q (P q (W q (X ch ))) WITH sp =F SG (σ z (K sp ×Q sp )) Among them, W q and W k is the convolutional layer, σ q , and σ z Represent the reshaping operation, P q and P k is the dimension transformation operation, F GM represents the global average operation, F SM is the softmax function, F SG is the sigmoid activation function; The final output of the PSSA submodule is the spatial enhanced feature map V sp =P v (s v (W v (X ch ))) X sp =W out (s out (P out (V sp ⊙Z sp ))) Among them, W v and W out is the convolutional layer; P v and P out is the dimension transformation operation; σ v and σ out represents a reshape operation; ⊙ represents an element-wise multiplication.

4. The pulmonary embolism auxiliary diagnosis system based on heterogeneous cross-modal fusion as claimed in claim 1, characterized in that: The multi-scale feature fusion module specifically includes: Assume that the intermediate features of the two parallel encoders at the same level are Attention Weight The calculation formula is as follows: w =σ(W se (X+Y)+W aspp (X+Y)+W 3dconv (X+Y)) Among them, W se (X+Y), W aspp (X+Y) and W 3dconv (X+Y) represents the output of SE module, ASPP module and 3D convolution respectively, σ represents Sigmoid activation function; Squeeze and Excitation (SE) module: The SE module enhances the feature representation of the channel dimension by adaptively weighting each channel, thereby increasing the focus on important features. The specific calculation is as follows: W se (X+Y)=(X+Y)·f se f se =σ(W se1 f1) f1=ReLU(W se2 (GlobalAvgPool(X+Y))) Where W se1 and W se2 is the learned weight matrix, GlobalAvgPool(·) represents the global average pooling operation; Expanded Spatial Pyramid Pooling (ASPP) module: The ASPP module uses expanded convolutions of different scales to capture multi-scale contextual information and enhance the model's ability to represent features under different receptive fields. The specific calculation is as follows: W aspp (X+Y)=W aspp ·f aspp f aspp =Conv 1×1×1 (f aspp1 ) Z=Conv 1×1×1 (GlobalAvgPool(X+Y)) Among them, r1, r2 and r3 represent the expansion rates set to 6, 12 and 18 respectively; 3D convolution module: The 3D convolution module enhances feature extraction in the spatial dimension, helping to capture complex spatial information and contextual relationships; the specific calculation is as follows: W 3dconv (X+Y)=Conv 1×1×1 (V)+(X+Y) V=GELU(BN(Conv 3×3×3 (X+Y))) In order to integrate features from different branches and introduce adaptive weights, a Sigmoid activation function is applied to map the input features to the range of 0 to 1; the calculated attention weights are used to weight the input features, which are then fused to obtain the final output feature map. The specific calculation is as follows: X'=X·w Y'=Y·w Z=Concat(X'+Y') To ensure that the features of each level are effectively utilized in pulmonary embolism detection, skip connections are adopted; each encoder layer contains a residual unit with a 3D convolution layer of specified size, an adaptive maximum pooling layer, and instance normalization; the processed features of each level are spatially reduced by half and connected with the results of multi-scale feature fusion to facilitate subsequent pulmonary embolism detection tasks.

5. The pulmonary embolism auxiliary diagnosis system based on heterogeneous cross-modal fusion as claimed in claim 1, characterized in that: The EHR module specifically includes: (1) Feature transformer Let the input be a one-dimensional feature vector Where B is the batch size and D is the feature dimension. The processing flow of the feature transformer is described as follows: In the Feature transformer, each feature block consists of a fully connected layer (FC), a batch normalization layer (BN), and a GLU layer; the input features are first linearly transformed through the fully connected layer, then batch normalized, and finally nonlinearly transformed through the GLU layer; this combination provides powerful expressive power for the features; the feature transformer contains multiple GLU layers, of which the first two layers are shared in all decision steps to ensure the consistency of the feature transformation process; these shared GLU layers maintain the unity of capturing global features by applying the same parameters in different decision steps; the last two GLU layers depend on the decision step, allowing personalized adjustments based on the feature state of each decision point; this design enriches the feature representation and provides flexibility between different decision steps; the outputs of these feature blocks are combined with normalization through residual connections, where the normalization factor is This normalization maintains the stability of feature variance during the learning process, avoiding drastic fluctuations in network training, thereby enhancing the stability of the overall training; this structure imitates the decision tree, which helps to effectively integrate and synthesize input feature information and capture complex feature interaction relationships; (2) Attentive transformer The Attentive Transformer consists of four main components: a fully connected layer (FC), a batch normalization layer (BN), a priori scaling layer, and a sparse maximization layer. This module enhances the learning ability of the model and improves the efficiency of feature selection through effective feature selection and masking mechanisms. In the Attentive transformer, the batch normalized features are element-wise multiplied with the prior scaling factor P[i], which is defined as follows: Among them, γ is a relaxation parameter, M[j] represents the mask feature from the previous stage; The core task of the Attentive Transformer is to determine the mask layer of the current step based on the result of the previous step; important features are learned through masks is sparsely selected; the masking mechanism ensures that the model's learning ability is focused on relevant features, thereby improving parameter efficiency; the attention transformer generates a mask from the processed features through the following formula: M[i]=sparsemax(P[i-1]*h i (a[i-1])) Among them, h i is a trainable function consisting of a fully connected layer (FC layer) and a batch normalization layer (BN layer), P[i-1] is the previous scaling factor from the previous stage, and sparsemax(·) is the activation function used to generate the sparse mask, defined as: sparsemax(x)=max(0,x-τ) Where τ is a threshold value, calculated as follows: here, Represents x j -The positive part of τ, k is the number of features to be retained; in the sparse maximization process, τ is a minimum value to ensure that exactly k features are retained.

6. The pulmonary embolism auxiliary diagnosis system based on heterogeneous cross-modal fusion as claimed in claim 1, characterized in that: The fusion module specifically includes: For the image feature x i and the table features y i , calculate the matching score β j,i and ρ j,i To evaluate the similarity between image features and table features; these scores are calculated by the dot product of the query vector and the key vector, and then normalized using the Softmax function, as follows: Among them, q1(x i )、k1(x j ),q2(y i ) and k2(y j ) represent the image features x after linear transformation. i and the table features y i The query vector and key vector of ; In order to achieve the calculation of the above matching scores, three different fusion modules are designed; Softmax Fusion Module: Inspired by the bidirectional multimodal attention mechanism, a Softmax fusion module is proposed, which realizes effective feature interaction by utilizing the matching score formula; specifically, it extracts the input image feature X from the image feature X. I and table feature X T The query vector (Q), key vector (K) and value vector (V) are generated in the following way: Q I ,K I ,V I =LP(Norm(X I )) Q T ,K T ,V T =LP(Norm(X T )) Among them, LP(·) represents the linear projection operation, Norm(·) represents layer normalization; the definition of multimodal attention fusion is: F I =Attention(Q I ,K I ,V I )+Attention(Q I ,K T ,V T ) F T =Attention(Q T ,K T ,V T )+Attention(Q T ,K I ,V I ) Here Attention(Q,K,V) consists of two matrix multiplications and a scaled Softmax operation: Where T represents the matrix transpose, d k is a scaling hyperparameter; the attention results are combined with the original features through residual connections to prevent information loss and stabilize the training process: x I =F I +X I x T =F T +X T The fused image features and table features are then input into a fully connected layer to generate the final prediction results of pulmonary embolism; NonlinearFusion Module: The nonlinear fusion module introduces the kernel function transformation φ(·) to perform nonlinear transformation on the query vector and the key vector, so that the model can capture more complex feature relationships beyond the linear matching score formula. The specific calculation is as follows: Where φ(·)=elu(x)+1 represents the kernel function; inspired by the linear attention mechanism, the definition of multimodal attention fusion is: This kernel function transformation enhances the model’s ability to capture complex nonlinear relationships, enabling the nonlinear fusion module to better understand the interactions between multimodal data. Through residual connections, the attention results are combined with the original features, and the fused features are input into the fully connected layer to generate the final pulmonary embolism prediction. Hybrid Fusion Module: The hybrid fusion module combines features from the Softmax and nonlinear fusion modules; generates query (Q), key (K), and value (V) vectors, and applies kernel function transformation to the query and key vectors of each modality; however, unlike the nonlinear fusion module, the hybrid fusion module retains the multiplication order of the Softmax fusion module, and the multimodal attention fusion formula is: In F I and F T In the formula, the first term represents the self-attention mechanism in the image modality and the table modality respectively, while the second term represents the cross-modal attention mechanism; by combining the kernel function transformation with the traditional attention mechanism, the hybrid fusion module is able to handle linear and nonlinear relationships, provide flexibility and capture more comprehensive cross-modal interactions; the attention results are integrated with the original features through residual connections, and the fused features are input into the fully connected layer to generate the final prediction of pulmonary embolism.

7. A pulmonary embolism auxiliary diagnosis method based on heterogeneous cross-modal fusion based on the pulmonary embolism auxiliary diagnosis system based on heterogeneous cross-modal fusion as described in claims 1-6, characterized in that: The method specifically includes: S1: A feature extraction model combining Swin Transformer and improved 3D CNN is established; the hybrid architecture uses Swin Transformer and CNN in parallel to extract more comprehensive features at different levels; in order to improve the network's ability to capture global and local features, a multi-scale feature fusion MSFF module is introduced; this module fuses the features extracted by Swin Transformer and 3D CNN at different scales; the feature extraction capability of 3D CNN is further enhanced by adding a polarized self-attention PSA module; in the overall architecture, multiple layers of residual convolutional layers and convolutional layers are used to ensure that the features from the two branches can be effectively fused and improve the training effect of the network; these convolutional layers help to fuse and downsample the features, thereby optimizing the model's performance in feature extraction and integration; S2: Utilizes the polarized self-attention module so that the model can efficiently process high-resolution 3D input features with minimal computational overhead while effectively capturing long-range dependencies for accurate voxel-level predictions; introduces the polarized self-attention PSA module; this module consists of two components: the polarized channel self-attention PCSA submodule and the polarized spatial self-attention PSSA submodule; S3: Using the multi-scale feature fusion module, we integrate the channel feature extraction from the squeeze and excitation SE module, the multi-scale feature extraction from the dilated spatial pyramid pooling ASPP module, and the spatial feature extraction from 3D convolution; this method makes the feature fusion of the Swin Transformer and 3D CNN encoder more effective at each level; S4: In order to effectively extract features from electronic health record (EHR) data, the six tabular files were first systematically standardized. This process involved removing features with zero variance and standardizing the remaining features by subtracting the mean and dividing by the standard deviation. This normalization ensured that all features were on comparable scales, facilitating meaningful comparisons. Subsequently, multiple tabular datasets were merged into a unified table based on the patient index number. Feature and attention transformers (FATs) were proposed. This module integrates a feature transformer, an attention transformer, and a new branch. The feature transformer is responsible for basic feature transformation, using a gated linear unit (GLU) to extract features and perform nonlinear transformations on them. This design enables the model to effectively capture complex relationships between features. The attention transformer applies an attention mechanism to weight features to highlight their importance. The new branch consists of a fully connected layer and an adaptive pooling layer. The combination of the fully connected layer and the adaptive pooling enables the branch to effectively aggregate global context while dynamically adjusting the feature space. S5: Utilize fusion modules, including three different multimodal fusion modules (MAFM); during the implementation process, key features are first extracted from the imaging module and the EHR module; then, these features are input into the MAFM for comprehensive fusion to optimize the diagnostic effect and enhance the complementarity between the modalities.

8. A computer device, characterized in that: The computer device includes a memory and a processor, the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of the pulmonary embolism auxiliary diagnosis method based on heterogeneous cross-modal fusion as described in claim 7.

9. A computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the processor executes the steps of the method for auxiliary diagnosis of pulmonary embolism based on heterogeneous cross-modal fusion as claimed in any one of claim 7.

10. An information data processing terminal, characterized in that: The information data processing terminal is used to implement the pulmonary embolism auxiliary diagnosis system based on heterogeneous cross-modal fusion as described in any one of claims 1-6.

Images