Multi-modal data processing method and device based on attention mechanism, equipment and medium

Abstract

The invention discloses a multi-modal data processing method and device based on an attention mechanism, equipment and a medium. Living habits of a user and living and medical characteristics of genes are combined on the basis of image data and electronic medical records. The method is advantaged in that the problem of difficult fusion of heterogeneous multi-source data is solved through uniform characterization of features of four-modal data, adaptive adjustment of attention weight of each modal is carried out through the entity association relationship in the entity relationship knowledge graph, and fusion precision of full-modal fusion features is improved. Through the anti-illusion prediction model, disease type and probability prediction is carried out for individual differences (user tags and full-modal fusion features) of target users, and the accuracy of disease prediction results is improved. The multi-modal data processing method can be applied to user body evaluation in the financial field and disease type prediction in the medical field so as to improve the risk prediction accuracy in the financial field and the disease type prediction accuracy in the medical field.

Description

Technical Field

[0001] This invention relates to the field of intelligent decision-making technology, and is applied to the fields of financial risk prediction and medical diagnosis. In particular, it relates to a multimodal data processing method, apparatus, computer equipment, and computer-readable storage medium based on an attention mechanism. Background Technology

[0002] Currently, when using AI models in the medical or financial fields to predict user ailments, they often employ single-modality or simple data aggregation methods, resulting in highly fragmented data. They fail to consider the deep correlations between different modalities and do not effectively integrate modal data such as user lifestyle habits, emotional state, and family background for ailment prediction. This reduces the accuracy of medical AI models in predicting rare diseases or complications. Therefore, improving the accuracy of ailment prediction results has become an urgent problem to be solved. Summary of the Invention

[0003] This invention provides a multimodal data processing method, apparatus, computer device, and computer-readable storage medium based on an attention mechanism to solve the technical problem of data fusion distortion in current multimodal models.

[0004] Firstly, an attention-based multimodal data processing method is provided, including:

[0005] The original multimodal array of the target user is preprocessed, and features are extracted from the preprocessed original multimodal array to generate a structured feature vector group. The original multimodal array includes image data, electronic medical records, genetic data, and behavioral data.

[0006] Based on the preset entity relationship knowledge graph and the structured feature vector group, the attention weight of each modality data in the original multimodal array is determined;

[0007] Based on the attention weights of each modality data, hierarchical attention fusion is performed on the structured feature vector to generate full-modality fusion features;

[0008] Based on preset user tags, anti-hallucination prediction models, and the full-modal fusion features, the disease type and its corresponding probability of the target user are predicted.

[0009] Secondly, a multimodal data processing device based on an attention mechanism is provided, comprising:

[0010] The structural feature extraction module is used to preprocess the original multimodal array of the target user and extract features from the preprocessed original multimodal array to generate a structured feature vector group. The original multimodal array includes image data, electronic medical records, genetic data and behavioral data.

[0011] The modality weight determination module is used to determine the attention weight of each modality data in the original multimodal array based on the preset entity relationship knowledge graph and the structured feature vector group;

[0012] The modal feature fusion module is used to perform hierarchical attention fusion on the structured feature vector based on the attention weights of each modal data to generate full-modal fusion features;

[0013] The user symptom prediction module is used to predict the symptom type and its corresponding probability of the target user based on preset user tags, anti-hallucination prediction model and the full-modal fusion features.

[0014] Thirdly, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described attention-based multimodal data processing method.

[0015] Fourthly, a computer-readable storage medium is provided, which stores a computer program that, when executed by a processor, implements the steps of the above-described attention-based multimodal data processing method.

[0016] In the aforementioned multimodal data processing method, apparatus, computer equipment, and computer-readable storage medium based on attention mechanisms, this application first collects multimodal data such as image data, genetic data, electronic medical records, and behavioral data. Based on medical history (i.e., image data and electronic medical records), and combined with the user's lifestyle habits and genetic medical characteristics, it predicts symptoms from multiple dimensions. Furthermore, it uses a unified representation of the features of the four modalities to solve the problem of heterogeneous and difficult fusion of multi-source data. It determines the relationships between entities in each modality through an entity relationship knowledge graph and adaptively adjusts the attention weights of each modality based on these relationships, improving the fusion accuracy of the full-modal fusion features. Finally, through an anti-hallucination prediction model, it predicts the type and probability of symptoms based on individual differences of the target user (user tags and full-modal fusion features), improving the accuracy of the symptom prediction results. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating a multimodal data processing method based on an attention mechanism according to an embodiment of the present invention.

[0019] Figure 2 This is a schematic diagram of a multimodal data processing architecture based on an attention mechanism in one embodiment of the present invention;

[0020] Figure 3 yes Figure 1 A schematic diagram of a specific implementation method for step S30;

[0021] Figure 4 This is a schematic diagram of a multimodal data processing device based on an attention mechanism according to an embodiment of the present invention;

[0022] Figure 5 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention. Detailed Implementation

[0023] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0024] The multimodal data processing method based on the attention mechanism involved in the embodiments of this application is mainly applied to computer devices. The multimodal data processing generation device based on the attention mechanism can be a PC, a portable computer, a mobile terminal, or other devices with display and processing functions.

[0025] The following detailed description of some embodiments of this application is provided in conjunction with the accompanying drawings. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0026] Reference Figure 1 , Figure 1 This is a flowchart illustrating a multimodal data processing method based on an attention mechanism provided in this application.

[0027] like Figure 1As shown, this application provides a multimodal data processing method based on an attention mechanism, which includes steps S10 to S40.

[0028] In this embodiment, the attention-based multimodal data processing method includes the following steps:

[0029] Step S10: Preprocess the original multimodal array of the target user, and extract features from the preprocessed original multimodal array to generate a structured feature vector group. The original multimodal array includes image data, electronic medical records, genetic data, and behavioral data.

[0030] In this embodiment, the image data is DICOM format CT / MRI three-dimensional volumetric data; the electronic medical record is free text containing the chief complaint, present illness history, and laboratory report paragraphs; the genetic data is a VCF file containing SNPs / InDels and annotations; and the behavioral data is a JSON stream of heart rate, steps, and sleep stages sampled at 1Hz by a smart bracelet. The collected raw data can be uploaded to a preprocessing server via HTTPS+FHIRBundle, sorted by timestamp, and stored in a Redis Stream "raw_queue" for preprocessing.

[0031] The system comprehensively considers multiple dimensions of data, including lesion features in medical images corresponding to original CT / MRI images, medical event logic chains corresponding to unstructured text medical records, genetic variations corresponding to original gene testing data, and lifestyle data corresponding to time-series data collected by wearable devices. It also takes into account the user's physical data, examination results, and lifestyle habits.

[0032] Specifically, such as Figure 2 As shown, feature data from images (CT / MRI) are extracted based on convolutional neural networks (CNNs). This allows for the identification and quantification of lesion features in medical images through CNNs, and the transformation of pixel-level data into structured feature vectors that can be quantified and analyzed.

[0033] By using a transformer to extract text data from electronic medical records and based on a BiLSTM-CRF entity extraction network, event chains Ztext (such as the complete process of chief complaint → examination → diagnosis → treatment) can be extracted from unstructured text medical records. Specifically, a bidirectional LSTM network and a conditional random field model can be used to construct a temporal logical chain of medical events, integrating fragmented text into a semantic sequence with medical significance, thereby solving the problem of discreteness of text data.

[0034] By using the Evo large model to transform gene detection data, in order to establish the association between genotype and phenotype, the gene association network Zgene (i.e., SNP site → target gene) is extracted from the raw gene detection data based on the pgBoost algorithm. Then, key SNP mutations are identified through site screening algorithms, and a gene regulatory network map is constructed to uncover the biological link between genetic variation and disease manifestation.

[0035] Data on sleep patterns, privacy habits (such as breakfast and dinner times), exercise frequency, smoking, drinking, and dietary habits were collected and structured using smart cameras and wearable devices. To unify multi-source, heterogeneous lifestyle data, the time-series data collected by wearable devices was normalized using the FHIR standard to obtain standardized behavioral features Zcustom. Then, the Medical Information Interchange Standard (FHIR) was used to perform spatiotemporal alignment and format conversion of the behavioral data, achieving comparability and integration of data collected from different devices.

[0036] By preprocessing data from four dimensions—spatial feature processing (images), semantic feature processing (text), regulatory network processing (genes), and temporal feature processing (behavior)—the above methods achieve unified representation of four types of heterogeneous data: CT images (spatial data), text records (semantic data), gene sequences (network data), and behavioral signals (temporal data). This eliminates the modal gap and generates a set of structured feature vectors that can be recognized by the prediction model.

[0037] In one embodiment, the step of extracting features from the preprocessed original multimodal array to generate a structured feature vector set includes:

[0038] Based on the spherical coordinate system transformation rules, spatial features are extracted from the image data to obtain structured image feature vectors;

[0039] Based on the UMLS CUI encoding rules, a semantic chain is constructed on the electronic medical record to obtain a text semantic event chain;

[0040] Based on the pgBoost path selection rules, the gene data is analyzed to obtain the gene regulatory network.

[0041] Based on the FHIR segmentation aggregation rule, the behavioral data is standardized by behavioral temporal features to obtain a behavioral temporal feature matrix;

[0042] The structured image feature vector, text semantic event chain, gene regulatory network, and behavioral temporal feature matrix are combined to form the structured feature vector.

[0043] In this embodiment, image data, electronic medical records, genetic data, and behavioral data can be preprocessed and feature extracted in parallel.

[0044] The original DICOM format image data was resampled to 1mm isoctex pixels. Then, with the lesion center as the origin and a radius of 60mm, 256 directions were uniformly sampled. The sampled data was then transformed into a spherical coordinate system. Finally, 3D-CNN (ResNet-50) was used to extract the coordinate-transformed image data, resulting in a 2048-dimensional structured image feature vector.

[0045] Using spaCy-BioNER, entities such as diseases, medications, and symptoms are extracted from electronic medical records, and each entity is mapped to a UMLS CUI. Then, a text semantic event chain is constructed using CUI as nodes and co-occurrence frequency as edges.

[0046] Based on pgBoost pathway screening, pathways with P-value < 1e-5 and functional score > 0.8 were screened from VCF. Gene data from the screened pathways were used as network nodes and regulatory weights were used as edges to construct a gene regulation network.

[0047] The behavior JSON is segmented and aggregated using FHIR Observation at preset time intervals (e.g., every 5 minutes), and the mean, variance, and frequency domain energy of the aggregated behavior data are calculated. A 128×N behavior time series feature matrix is ​​then constructed based on the calculation results.

[0048] Therefore, based on the above feature combination, a four-element structured feature vector group is obtained.

[0049] Specifically, the original CT / MRI image data containing spatial coordinate information of the target user undergoes spatial coordinate transformation, that is, the Cartesian coordinates (x, y, z) of the lesion in the image are converted into spherical coordinates (r, θ, φ). The spherical coordinates are the outermost surface corresponding to the location of the lesion, such as the spherical coordinates of a nodule in the lower lobe of the right lung being (120, 45, 30). Then, feature extraction is performed on the transformed image data. Specifically, a 3D-CNN network can be used to extract 128-dimensional radiomics features (including texture, shape, and wavelet features) from the transformed image data, resulting in a 131-dimensional (128-dimensional radiomics features + 3-dimensional spherical coordinates) structured image feature vector Zimage.

[0050] Event chains are constructed based on the target user's electronic medical records (such as unstructured text like outpatient / inpatient records). These event chains are used to construct decision paths, such as a chief complaint of chest pain, undergoing a coronary CT scan, and being diagnosed with coronary heart disease. Specifically, a BiLSTM-CRF model can be used to parse the electronic medical records, obtaining a time-series event chain of chief complaint → examination → diagnosis → treatment. Then, the terms in the event-series chain are standardized. Specifically, medical terms can be mapped according to UMLS CUI encoding rules, such as "chest pain" → ICD-10 encoding R07.9, resulting in a 256-dimensional textual semantic event chain Ztext (Clinical-BERT encoded vector). After obtaining the textual semantic event chain and the structured image feature vector Zimage, image features can be associated with textual descriptions (such as "right lower lobe nodule") by matching corresponding coordinates, or calcification descriptions can be associated with "BI-RADS 4 categories" terms in the case. This involves an anatomical mapping of spatial coordinates to terms, completing the initial combination of image and textual features, thereby mining the correlation between image and textual features.

[0051] SNP screening is performed on the raw gene sequencing files of the target users. Specifically, pathogenic SNP sites (i.e., p-value < 0.001) can be screened using the pgBoost algorithm, and then a regulatory network is constructed based on these SNP sites. This regulatory network is used to analyze pathogenic molecular mechanisms, such as how SNP rs1042522 leads to TP53 protein inactivation, thereby increasing the risk of lung cancer. Specifically, an expression regulatory pathway from SNP to target gene to protein can be generated based on the STRING database, resulting in a 200-dimensional vector gene regulatory network Zgene containing the relationship between SNP association strength and target gene. Gene risk prediction is then performed using this gene regulatory network. After obtaining the textual semantic event chain corresponding to the electronic medical record and the gene regulatory network corresponding to the gene data, causal modeling based on the regulatory network and radiomics is performed to mine the correlation between gene features and imaging features.

[0052] Behavioral data standardization is performed on raw streaming data collected from wearable devices (such as accelerometer / GPS signals) of the target user. For example, the data is aggregated in segments according to a 15-minute window and the FHIR R4 specification. Temporal feature encoding is then performed on the standardized data, and the behavioral segments are used to quantify the impact of lifestyle habits. Specifically, behavioral habit features (such as sleep cycles and exercise frequency) can be extracted from the raw streaming data using an LSTM network to obtain a 100-dimensional LSTM hidden layer feature matrix Zcustom. When obtaining the gene risk prediction results corresponding to the gene data and the temporal feature matrix representing lifestyle habit features, the disease probability is dynamically adjusted based on the behavioral temporal features, thereby associating the gene risk prediction results with the impact of lifestyle habits (e.g., smoking behavior increases the risk weight of lung cancer).

[0053] The structured feature vector group [Zimage, Ztext, Zgene, Zcustom] is obtained based on the combination of feature vectors of each modality.

[0054] The above approach first structures and standardizes the four modalities of data, then converts the differential data from the four different dimensions into a standardized matrix with a unified feature representation. This eliminates the semantic gap between the modalities and addresses the issue of diverse medical data sources. Furthermore, it mines the correlations between the data from different modalities to improve the accuracy of disease type prediction.

[0055] Step S20: Based on the preset entity relationship knowledge graph and the structured feature vector group, determine the attention weight of each modality data in the original multimodal array;

[0056] In this embodiment, a search is performed on the entity relationship knowledge graph based on structured feature vector groups, and the retrieved entity relationship list is dynamically encoded to obtain standardized entity feature vectors. Weights are then calculated based on the standardized entity feature vectors, a multilayer perceptron, and a normalized exponential function to obtain a four-modal attention weight vector (image weight, text weight, gene weight, and behavior weight).

[0057] Using the entity IDs in the structured feature vector set [Zimage, Ztext, Zgene, Zcustom] as keys, a query is performed in the pre-defined entity relationship knowledge graph Neo4j to obtain a list of entity relationships. The entities in the structured feature vector set are then embedded into this list, and concatenated with the structured feature vector set to obtain a 512-dimensional standardized entity feature vector. This standardized entity feature vector is then input into a two-layer MLP and a normalized exponential function Softmax to obtain a four-modal attention weight vector.

[0058] Step S30: Based on the attention weights of each modality data, perform hierarchical attention fusion on the structured feature vector to generate full-modality fusion features;

[0059] In this embodiment, hierarchical attention fusion includes bimodal fusion, trimodal interaction, and quadrumodal decision-making. The text semantic event chain and image feature vector are fused bimodally to obtain text-image fusion features. The text-image fusion features are then interacted with a gene regulatory network in a trimodal manner to obtain text-image-gene fusion features. Finally, the text-image-gene fusion features are combined with a behavioral temporal feature matrix for quadrumodal decision-making to obtain full-modal fusion features.

[0060] Step S40: Based on the preset user tags, the anti-hallucination prediction model, and the full-modal fusion features, predict the disease type of the target user and its corresponding probability.

[0061] In this embodiment, user tags are pre-acquired by retrieving the target user's corresponding disease type and user profile from the hospital information system, which serve as tag vectors. The tag vectors of the target users are then combined to redistribute the weights of the data across different modalities. Specifically, the full-modal fusion features and user tag vectors are input into the anti-hallucination prediction model to obtain the disease probability distribution and weight allocation matrix (4×C, where C is the disease category). The full-modal fusion features and the weight allocation matrix are then weighted and fused, i.e., weighted by category, to obtain the target user's disease type and its corresponding probability.

[0062] In one embodiment, step S40 includes:

[0063] Obtain the user tags of the target user, the user tags including relevant disease types and user profiles;

[0064] Based on the anti-hallucination prediction model, the full-modal fusion features, and the user tags, the disease type of the target user is predicted, and the weights of each modality data are assigned based on the analysis results to obtain a weight assignment matrix.

[0065] The full-modal fusion features and the weight allocation matrix are weighted and fused, and based on the fusion results and the disease type prediction results, the disease type of the target user and its corresponding probability are predicted.

[0066] In this embodiment, based on the full-modal fusion features, user tags used to represent individual user differences are further combined to predict the disease type and probability, and the weight matrix is ​​redistributed at the same time to generate an adaptive weight matrix that can be used for the next prediction. Through the prediction-interpretation-recalibration mechanism, the accuracy of the model prediction results is continuously improved.

[0067] Specifically, the first step is to read the target user's user tags. These user tags can be obtained from the original tag set of the hospital information system, including relevant disease types and user profiles (including but not limited to age, gender, body mass index, smoking history, family history, etc.).

[0068] In a specific embodiment, user tags can be generated in real time, and the user tags can be cleaned, such as filling in missing values, tag encoding and vectorization, and finally concatenated to obtain preprocessed user tags.

[0069] The anti-hallucination prediction model decomposes the full-modal fusion features into four 128-dimensional sub-vectors based on their source: image feature F_img, text feature F_txt, gene feature F_gene, and behavioral feature F_behav. Then, for each category k, the category features are calculated:

[0070] F_k=Σ_{m∈{img,txt,gene,behav}}W_realloc[m,k]·F_m

[0071] The 128-dimensional category feature vector {F_k} corresponding to the number of categories is obtained.

[0072] Finally, each F_k is input into the fully connected layer (1) of the corresponding category to obtain the logical value _k. Then, the final probability is obtained by passing the normalized exponential function (logical value _1...number of categories). Thus, the disease type and its probability are predicted.

[0073] This embodiment provides a solution implemented by a multimodal data processing method, apparatus, computer device, and computer-readable storage medium based on an attention mechanism. First, multimodal data such as image data, genetic data, electronic medical records, and behavioral data are collected. Based on medical history (i.e., image data and electronic medical records), and combined with the user's lifestyle habits and genetic medical characteristics, disease prediction is performed from multiple dimensions. Furthermore, the features of the four modalities are uniformly represented to solve the problem of heterogeneous data fusion. The relationships between entities in each modality are determined through an entity relationship knowledge graph, and the attention weights of each modality are adaptively adjusted according to these relationships to improve the fusion accuracy of the full-modal fusion features. Finally, an anti-hallucination prediction model is used to predict the type and probability of diseases based on individual differences of the target user (user tags and full-modal fusion features), improving the accuracy of the disease prediction results.

[0074] Please see Figure 3 , Figure 3 This is a schematic flowchart illustrating a specific implementation of step S30 provided in an embodiment of this application.

[0075] like Figure 3 As shown, step S30 specifically includes:

[0076] Step S31: Perform feature projection on the text semantic event chain, the structured image feature vector, and the gene regulatory network respectively to obtain text features, image features, and gene features.

[0077] Step S32: Based on the bidirectional attention mechanism, perform bimodal fusion on the text features and the image features to obtain text-image fusion features;

[0078] Step S33: Based on the cross-modal attention mechanism, perform trimodal interaction on the text image fusion features and the gene features to obtain text image gene fusion features;

[0079] Step S34: Based on the hierarchical attention aggregation mechanism and the attention weights corresponding to the behavioral data, perform four-modal decision-making on the text image gene fusion features and the behavioral temporal encoding corresponding to the behavioral temporal feature matrix to generate the full-modal fusion features.

[0080] In this embodiment, the text semantic event chain and the image feature vector are linearly projected to 256 dimensions respectively, and based on bidirectional cross attention, the projected text semantic event chain and the image feature vector are fused in a dual-modal manner to obtain text-image fusion features.

[0081] Based on cross-attention, the text-image fusion features are again interacted with the gene regulatory network in three modes to obtain the text-image gene fusion features.

[0082] Based on long short-term memory networks, the text image gene fusion features are encoded into 256-dimensional behavioral temporal codes. Then, based on hierarchical attention, the encoded behavioral temporal codes are aggregated with the behavioral temporal feature matrix to obtain 512-dimensional full-modality fusion features.

[0083] By using the above method, the four heterogeneous features are gradually aggregated first in a bimodal manner, then in a trimodal manner, and finally in a quadmodal manner, so as to achieve accurate weighting and hierarchical dimensionality reduction of the four-modal features, reduce computational difficulty, and improve fusion efficiency.

[0084] In one embodiment, the preprocessing of the original multimodal array of the target user includes:

[0085] Based on the entity relationship knowledge graph, calculate the KL divergence of each term in the original multimodal array in different modal data, and perform ambiguity detection on each term based on the KL divergence;

[0086] When ambiguous terms are detected in each term, semantic disambiguation is performed on the ambiguous terms based on the attention weights of each modality data in the entity relationship knowledge graph.

[0087] In this embodiment, before feature extraction, semantic disambiguation is performed on the terms in the original multimodal array based on the entity relationship knowledge graph, thereby avoiding semantic conflicts among the features of different modalities.

[0088] Specifically, terminology extraction is first performed on the original multimodal array, such as the free text "patchy shadows seen in the upper lobe of the right lung" in the DICOM imaging report, the chief complaint, present illness history, and test results paragraphs in the electronic medical record, the mutation annotation field "BRCA1 c.5266dupC" in the gene data, and the questionnaire text "smoking history: yes" in the behavioral data, etc.

[0089] The extracted terms are then segmented, specifically using a medical word segmenter or medical dictionary, employing a bidirectional maximum matching rule to segment the text for each modality. Simultaneously, image structure tags (such as "patchy shadow") and mutation symbols (such as "c.5266dupC") are extracted.

[0090] The segmentation results are mapped to the Unique Medical Language System Concept Unique Identifier (CUI). If a direct mapping is not possible, the original words are retained and marked as terms to be disambiguated. In this way, terminology standardization is completed, resulting in a term list T = {t1, t2, ..., t}, with each term accompanied by a source modality label (image / text / gene / behavior).

[0091] Based on the entity relationship knowledge graph (node ​​= unique identifier of concept, edge = synonym / superordinate / related relationship), KL divergence ambiguity detection is performed on the term list.

[0092] First, perform term-modal distribution statistics, that is, for each term t i The frequency of its occurrence in the four modes was counted to obtain the discrete distribution P. i (m). If t i If it does not appear in mode m, then the smoothness is ε = 1e-5.

[0093] Then, the KL divergence is calculated, i.e., t is calculated. i Relative entropy between mode pairs:

[0094] DKL(P i (Image)||P i (text)), DKL(P i (Image)||P i (Genes)... a total of 6 groups.

[0095] If any DKL value is greater than the threshold θ (e.g., 0.5), then mark t. i Ambiguous terms are identified, resulting in a set of ambiguous terms. And its corresponding KL divergence matrix D_ambig.

[0096] For each ambiguous term t i Search for all synonyms / superior nodes in the graph to obtain the candidate concept set C. i ={c1,c2,…}.

[0097] Calculate the association score between candidate concepts and the current modality:

[0098] score(cj)=Σ_{m∈{image, text, gene, behavior}}W_attn[m]·PMI(cj,m)

[0099] Where PMI is concept-modal mutual information, argmax(score) is the disambiguated concept, and t is updated. i The mapping is a unique identifier for the corresponding concept.

[0100] Replace the original word with the disambiguated concept in the original text; if the replacement causes a syntactic conflict, then fall back to the second-best candidate.

[0101] This yields a disambiguated term list T_disamb, with all ambiguous terms mapped to knowledge graph nodes.

[0102] In one embodiment, before determining the attention weights of each modality data in the original multimodal array based on the preset entity relationship knowledge graph and the structured feature vector group, the process includes:

[0103] The structured feature vector is subjected to feature decoding, CUI mapping, and encoding fusion to obtain a standardized entity feature vector;

[0104] Based on the entity relationship knowledge graph, the dynamic relationships between entities in the standardized entity feature vector are determined;

[0105] Based on the dynamic relationships, cross-modal association modeling is performed to obtain various entity relationship groups, and based on each entity relationship group, the entity relationship knowledge graph is generated.

[0106] In this embodiment, before calculating attention weights, the structured features of the four modal data are first unified into medical concepts. Then, the relationships between the modalities are mined, and a knowledge graph is generated based on the relationships to provide a basis for subsequent weight allocation and anti-hallucination prediction.

[0107] Specifically, feature decoding is performed on the structured feature vector set:

[0108] The 2048-dimensional image features are decoded by a deconvolution decoder to obtain a 256-dimensional low-dimensional representation.

[0109] Perform 128-dimensional entity embedding and 128-dimensional relation embedding on event triples of text features.

[0110] The regulatory network of gene features is aggregated into a graph convolutional network to obtain a 256-dimensional node vector.

[0111] By using a bidirectional long short-term memory network, the temporal matrix of behavioral data is decoded to obtain a 256-dimensional sequence vector.

[0112] The medical entity linker maps each vector to a concept unique identifier (CUI), and marks feature vectors that cannot be mapped as "unknown" while retaining the original vectors.

[0113] The four types of 256-dimensional vectors are concatenated to obtain a 512-dimensional standardized entity feature vector. The standardized entity feature vector is then transformed into a 512-dimensional unified space through a two-layer multilayer perceptron for dimensionality reduction, thus obtaining the standardized entity features.

[0114] Calculate the cosine similarity between any two entity vectors in the standardized entity features, and add a potential dynamic relationship to two entities with a similarity > 0.75.

[0115] The weights are assigned using the formula: Dynamic Edge Weight = Similarity × Prior Graph Weight. If there is no prior graph weight, then similarity is used as the weight.

[0116] Based on the dynamic relationships, conflict detection is performed on the entities. If the same entity has multiple relationships, it indicates a conflict, and the one with the highest weight is retained. This results in a dynamic relationship list: {Source Entity, Target Entity, Relationship Type, Dynamic Weight}.

[0117] The dynamic relationship list is grouped by source entity + relationship type, and the average weight within each group is taken as the entity relationship group weight to complete the entity relationship group aggregation.

[0118] Write entity relationship groups into the graph database in batches (add nodes or update edge weights) to generate an updated entity relationship knowledge graph (named with version number + graph data).

[0119] The entity relationship knowledge graph is divided into entities according to modality (i.e., image, text, gene, behavior). The attention weight of each modality is equal to the normalized result of the sum of the outgoing edge weights of all entities in that modality, thus completing the modality-level weight aggregation.

[0120] Before predicting the target user's symptom type and its corresponding probability based on preset user tags, an anti-hallucination prediction model, and the full-modal fusion features, the method further includes:

[0121] Based on the time-delay mutual information causal discovery algorithm, the original medical multimodal data is time-aligned and preprocessed, and the cross-modal causal relationships between the data are extracted from the aligned data to obtain the cross-modal causal map between the data.

[0122] Based on the preset dynamic gating mechanism and the cross-modal causal graph, the weights of each modal data in the prediction model to be trained are adjusted to obtain the target weights corresponding to each modal data.

[0123] Modal contradictions are constructed from raw medical multimodal data to generate adversarial multimodal samples with conflict labels;

[0124] Based on the target weights and the cross-modal causal graph, the model to be trained is subjected to adversarial training with causal regularization constraints to obtain the anti-hallucination prediction model.

[0125] In this embodiment, the medical diagnostic system suffers from an attentional imbalance problem when processing multimodal medical data. Clinical tests show that when CT images reveal ground-glass nodules in the lungs but the electronic medical record does not record related symptoms, the system's attention to the image features drops to less than 15%, leading to a 23% missed diagnosis of early-stage lung cancer. To address the distortion in multimodal model data fusion caused by the modal attentional imbalance, an anti-hallucination processing structure involving data preprocessing, dynamic gating mechanisms, and adversarial training is implemented. Specifically:

[0126] Firstly, in the data preprocessing stage, a time-delayed mutual information causal discovery algorithm and a dynamic Bayesian network were used to model cross-modal causal relationships. Causal links between behavioral data, text data, genetic data, and image features were updated at 0.3-second intervals.

[0127] The second stage is a dynamic gating mechanism that uses an adaptive weight allocation strategy driven by KL divergence. When a cross-modal conflict is detected, a triple verification mechanism is triggered: gradient freezing, historical case matching, and attention recalibration.

[0128] The third stage is the anti-hallucination training framework, which includes adversarial example generation and causal regularization. Modal inconsistency sample pairs are constructed, and a spurious-correlation attention penalty term is used in the loss function.

[0129] Through the above three stages, the raw multimodal data is obtained, a causal graph is generated by the causal discovery module, the weights are adjusted by the dynamic gating mechanism, and finally adversarial training and regularization are performed in the training framework.

[0130] Specifically, in the first stage of data preprocessing, a time-delay mutual information causal discovery algorithm is used to extract causal relationships between cross-modal data in the original multimodal data and construct a cross-modal causal graph to guide the modality weight allocation of subsequent modules.

[0131] Multimodal data includes four modalities: image data, electronic medical records, genetic data, and behavioral data. The causal graph represents the causal relationships between modalities using a directed graph (e.g., the causal edge weight for image → diagnosis conclusion is 0.7).

[0132] The modal feature vectors (such as image features, text features, gene features, and behavioral features) and cross-modal causal graphs (such as the causal strength of behavior → image) corresponding to the modal data extracted in the first stage are input into the dynamic gating mechanism. The second stage of dynamic gating, namely the adaptive weight allocation based on KL divergence, is used to quantify the differences in cross-modal feature distribution, detect potential conflicts, and dynamically adjust the modal fusion weights according to the real-time feature distribution differences to resolve potential conflicts.

[0133] Based on the KL divergence calculation formula, the distribution difference between pairwise modes is calculated:

[0134]

[0135] P(i) and P(j) are the probability distributions corresponding to the two modalities, respectively, where i is the index of each data point in each modality.

[0136] If the CT image distribution P CT Distribution of medical record text P text D KL If θ > 0, then determine the CT image distribution P. CT Distribution of medical record text P text The data is contradictory.

[0137] Understandably, an adaptive threshold can be set based on the distribution of historical data (e.g., θ = μ + 2σ, where μ is the average KL divergence and σ is the standard deviation).

[0138] When cross-modal conflicts are detected in the modal data, the gradient propagation data corresponding to the modal data with cross-modal conflicts is frozen;

[0139] Based on the fusion vectors corresponding to each modality of data, search and match in historical conflict cases to identify historical similar cases;

[0140] Based on the weight correction strategy or attention recalibration strategy of the historical similar cases, the weights of the modal data are dynamically adjusted to obtain the target weights.

[0141] When there are no cross-modal conflicts in the modal data, the weights of each modal data are corrected based on the cross-modal causal graph to obtain the target weights of each modal data.

[0142] If there is a contradiction between the two modalities, a triple verification mechanism is triggered: first, the gradient propagation of the disputed modality is frozen; then, historical similar cases are used for pattern matching; and finally, the decision is corrected through cross-modal attention recalibration.

[0143] When calculating the weights of each modality, the dynamic gating module uses the causal intensity value extracted from the causal graph as the initial weight benchmark.

[0144] If the causal weight of CT images for lung cancer diagnosis in the causal graph is 0.7 (far higher than 0.1 in electronic medical record text), the gating mechanism will default to strengthening the decision-making dominance of the image modality.

[0145] The causal strength value α in the causal graph causal The initial values ​​for the gating weights are then dynamically adjusted using KL divergence.

[0146] Weight final =α causal +γ·D KL (P||Q)

[0147] Wherein, γ is the KL divergence adjustment coefficient, used to balance the differences between causal priors and real-time features.

[0148] When a difference in feature distribution between modes is detected (KL divergence exceeds the threshold), the system calls the causal graph to verify the causal credibility of the conflicting modes.

[0149] If a CT scan shows lung nodules but the medical record does not describe any symptoms, then:

[0150] 1) Based on the strong causal link of "image → diagnosis" in the causal graph (weight 0.8 vs text 0.05), the image modality is given priority.

[0151] 2) Retrieve similar historical cases (nodules on images but not recorded in the text) to correct the current diagnosis.

[0152] When retrieving similar historical cases, priority is given to selecting historical samples that match the current causal graph structure:

[0153] Construct adversarial examples of modal conflict data based on real samples (i.e., original multimodal data, text + images + numerical values, and their labeled data).

[0154] Among them, real samples can be CT images, electronic medical record texts, vital sign time series data, and real disease type labels in medical scenarios.

[0155] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0156] In one embodiment, an attention-based multimodal data processing apparatus is provided, which corresponds one-to-one with the attention-based multimodal data processing methods described in the above embodiments. For example... Figure 4 As shown, the attention-based multimodal data processing device 400 specifically includes:

[0157] The structural feature extraction module 410 is used to preprocess the original multimodal array of the target user and extract features from the preprocessed original multimodal array to generate a structured feature vector group. The original multimodal array includes image data, electronic medical records, gene data and behavioral data.

[0158] The modality weight determination module 420 is used to determine the attention weight of each modality data in the original multimodal array based on the preset entity relationship knowledge graph and the structured feature vector group;

[0159] The modal feature fusion module 430 is used to perform hierarchical attention fusion on the structured feature vector based on the attention weights of each modal data to generate full-modal fusion features;

[0160] The user symptom prediction module 440 is used to predict the symptom type and its corresponding probability of the target user based on preset user tags, anti-hallucination prediction model and the full-modal fusion features.

[0161] Furthermore, the structural feature extraction module 410 is also used for:

[0162] Based on the spherical coordinate system transformation rules, spatial features are extracted from the image data to obtain structured image feature vectors;

[0163] Based on the UMLS CUI encoding rules, a semantic chain is constructed on the electronic medical record to obtain a text semantic event chain;

[0164] Based on the pgBoost path selection rules, the gene data is analyzed to obtain the gene regulatory network.

[0165] Based on the FHIR segmentation aggregation rule, the behavioral data is standardized by behavioral temporal features to obtain a behavioral temporal feature matrix;

[0166] The structured image feature vector, text semantic event chain, gene regulatory network, and behavioral temporal feature matrix are combined to form the structured feature vector.

[0167] Furthermore, the modal feature fusion module 430 includes:

[0168] The feature extraction unit is used to perform feature projection on the text semantic event chain, the structured image feature vector, and the gene regulatory network respectively to obtain text features, image features, and gene features.

[0169] A dual-modal fusion unit is used to perform dual-modal fusion of the text features and the image features based on a bidirectional attention mechanism to obtain text-image fusion features;

[0170] The three-modal fusion unit is used to perform three-modal interaction on the text image fusion features and the gene features based on a cross-modal attention mechanism to obtain text image gene fusion features;

[0171] The four-modal fusion unit is used to perform four-modal decision-making on the text image gene fusion features and the behavioral temporal encoding corresponding to the behavioral temporal feature matrix based on the hierarchical attention aggregation mechanism and the attention weights corresponding to the behavioral data, so as to generate the full-modal fusion features.

[0172] Furthermore, the user symptom prediction module 440 includes:

[0173] The user tag acquisition unit is used to acquire the user tags of the target user, the user tags including relevant disease types and user profiles;

[0174] The weight dynamic allocation unit is used to predict the disease type of the target user based on the anti-hallucination prediction model, the full-modal fusion features and the user tags, and to allocate the weights of each modality data based on the analysis results to obtain a weight allocation matrix.

[0175] The symptom type prediction unit is used to perform weighted fusion of the full-modal fusion features and the weight allocation matrix, and predict the symptom type and its corresponding probability of the target user based on the fusion result and the symptom type prediction result.

[0176] Furthermore, the structural feature extraction module 410 is also used for:

[0177] Based on the entity relationship knowledge graph, calculate the KL divergence of each term in the original multimodal array in different modal data, and perform ambiguity detection on each term based on the KL divergence;

[0178] When ambiguous terms are detected in each term, semantic disambiguation is performed on the ambiguous terms based on the attention weights of each modality data in the entity relationship knowledge graph.

[0179] Furthermore, the attention-based multimodal data processing device 400 specifically includes a knowledge graph generation module, used for:

[0180] The structured feature vector is subjected to feature decoding, CUI mapping, and encoding fusion to obtain a standardized entity feature vector;

[0181] Based on the entity relationship knowledge graph, the dynamic relationships between entities in the standardized entity feature vector are determined;

[0182] Based on the dynamic relationships, cross-modal association modeling is performed to obtain various entity relationship groups, and based on each entity relationship group, the entity relationship knowledge graph is generated.

[0183] Furthermore, the attention-based multimodal data processing device 400 also includes a prediction model generation module, used for:

[0184] Based on the time-delay mutual information causal discovery algorithm, the original medical multimodal data is time-aligned and preprocessed, and the cross-modal causal relationships between the data are extracted from the aligned data to obtain the cross-modal causal map between the data.

[0185] Based on the preset dynamic gating mechanism and the cross-modal causal graph, the weights of each modal data in the prediction model to be trained are adjusted to obtain the target weights corresponding to each modal data.

[0186] Modal contradictions are constructed from raw medical multimodal data to generate adversarial multimodal samples with conflict labels;

[0187] Based on the target weights and the cross-modal causal graph, the model to be trained is subjected to adversarial training with causal regularization constraints to obtain the anti-hallucination prediction model.

[0188] Specific limitations regarding the attention-based multimodal data processing device can be found in the limitations of the attention-based multimodal data processing method described above, and will not be repeated here. Each module in the aforementioned attention-based multimodal data processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0189] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 5 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external clients via a network connection. When the computer program is executed by the processor, it implements the functions or steps of a multimodal data processing method based on an attention mechanism.

[0190] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps:

[0191] The original multimodal array of the target user is preprocessed, and features are extracted from the preprocessed original multimodal array to generate a structured feature vector group. The original multimodal array includes image data, electronic medical records, genetic data, and behavioral data.

[0192] Based on the preset entity relationship knowledge graph and the structured feature vector group, the attention weight of each modality data in the original multimodal array is determined;

[0193] Based on the attention weights of each modality data, hierarchical attention fusion is performed on the structured feature vector to generate full-modality fusion features;

[0194] Based on preset user tags, anti-hallucination prediction models, and the full-modal fusion features, the disease type and its corresponding probability of the target user are predicted.

[0195] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0196] Based on the spherical coordinate system transformation rules, spatial features are extracted from the image data to obtain structured image feature vectors;

[0197] Based on the UMLS CUI encoding rules, a semantic chain is constructed on the electronic medical record to obtain a text semantic event chain;

[0198] Based on the pgBoost path selection rules, the gene data is analyzed to obtain the gene regulatory network.

[0199] Based on the FHIR segmentation aggregation rule, the behavioral data is standardized by behavioral temporal features to obtain a behavioral temporal feature matrix;

[0200] The structured image feature vector, text semantic event chain, gene regulatory network, and behavioral temporal feature matrix are combined to form the structured feature vector.

[0201] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0202] Feature projections are performed on the text semantic event chain, structured image feature vector, and gene regulatory network respectively to obtain text features, image features, and gene features;

[0203] Based on a bidirectional attention mechanism, the text features and the image features are fused in a dual-modal manner to obtain text-image fusion features;

[0204] Based on a cross-modal attention mechanism, the text-image fusion features and the gene features are subjected to trimodal interaction to obtain text-image gene fusion features;

[0205] Based on the hierarchical attention aggregation mechanism and the attention weights corresponding to the behavioral data, a four-modal decision is made on the text image gene fusion features and the behavioral temporal encoding corresponding to the behavioral temporal feature matrix to generate the full-modal fusion features.

[0206] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0207] Obtain the user tags of the target user, the user tags including relevant disease types and user profiles;

[0208] Based on the anti-hallucination prediction model, the full-modal fusion features, and the user tags, the disease type of the target user is predicted, and the weights of each modality data are assigned based on the analysis results to obtain a weight assignment matrix.

[0209] The full-modal fusion features and the weight allocation matrix are weighted and fused, and based on the fusion results and the disease type prediction results, the disease type of the target user and its corresponding probability are predicted.

[0210] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0211] Based on the entity relationship knowledge graph, calculate the KL divergence of each term in the original multimodal array in different modal data, and perform ambiguity detection on each term based on the KL divergence;

[0212] When ambiguous terms are detected in each term, semantic disambiguation is performed on the ambiguous terms based on the attention weights of each modality data in the entity relationship knowledge graph.

[0213] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0214] The structured feature vector is subjected to feature decoding, CUI mapping, and encoding fusion to obtain a standardized entity feature vector;

[0215] Based on the entity relationship knowledge graph, the dynamic relationships between entities in the standardized entity feature vector are determined;

[0216] Based on the dynamic relationships, cross-modal association modeling is performed to obtain various entity relationship groups, and based on each entity relationship group, the entity relationship knowledge graph is generated.

[0217] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0218] Based on the time-delay mutual information causal discovery algorithm, the original medical multimodal data is time-aligned and preprocessed, and the cross-modal causal relationships between the data are extracted from the aligned data to obtain the cross-modal causal map between the data.

[0219] Based on the preset dynamic gating mechanism and the cross-modal causal graph, the weights of each modal data in the prediction model to be trained are adjusted to obtain the target weights corresponding to each modal data.

[0220] Modal contradictions are constructed from raw medical multimodal data to generate adversarial multimodal samples with conflict labels;

[0221] Based on the target weights and the cross-modal causal graph, the model to be trained is subjected to adversarial training with causal regularization constraints to obtain the anti-hallucination prediction model.

[0222] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0223] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0224] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0225] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. An attention mechanism-based multi-modal data processing method, characterized in that, The method comprises the following steps: preprocessing the original multi-modal array of the target user, and extracting features from the preprocessed original multi-modal array to generate a structured feature vector group, wherein the original multi-modal array includes image data, electronic medical records, genetic data, and behavior data; determining the attention weights of each modality data in the original multi-modal array based on a preset entity relationship knowledge graph and the structured feature vector group; performing hierarchical attention fusion on the structured feature vector based on the attention weights of each modality data to generate a full-modal fusion feature; predicting the disease type and its corresponding probability of the target user based on a preset user label, an anti-illusion prediction model, and the full-modal fusion feature. 2.The attention mechanism based multi-modal data processing method of claim 1, wherein, The method of extracting features from the preprocessed original multi-modal array to generate a structured feature vector group comprises the following steps: performing spatial feature extraction on the image data based on a spherical coordinate system conversion rule to obtain a structured image feature vector; constructing a semantic chain based on a UMLS CUI coding rule to obtain a text semantic event chain; performing regulatory network analysis on the genetic data based on a pgBoost path screening rule to obtain a gene regulatory network; performing behavior time series feature standardization on the behavior data based on a FHIR segmentation aggregation rule to obtain a behavior time series feature matrix; combining the structured image feature vector, the text semantic event chain, the gene regulatory network, and the behavior time series feature matrix into the structured feature vector. 3.The attention mechanism based multi-modal data processing method of claim 2, wherein, The method of performing hierarchical attention fusion on the structured feature vector based on the attention weights of each modality data to generate a full-modal fusion feature comprises the following steps: performing feature projection on the text semantic event chain, the structured image feature vector, and the gene regulatory network respectively to obtain text features, image features, and gene features; performing dual-modal fusion on the text features and the image features based on a bidirectional attention mechanism to obtain text-image fusion features; performing three-modal interaction on the text-image fusion features and the gene features based on a cross-modal attention mechanism to obtain text-image-gene fusion features; performing four-modal decision on the text-image-gene fusion features and the behavior time series encoding corresponding to the behavior time series feature matrix based on a hierarchical attention aggregation mechanism and the attention weights corresponding to the behavior data to generate the full-modal fusion feature. 4.The attention mechanism based multi-modal data processing method of claim 1, wherein, The method of predicting the disease type and its corresponding probability of the target user based on a preset user label, an anti-illusion prediction model, and the full-modal fusion feature comprises the following steps: obtaining the user label of the target user, wherein the user label includes related disease types and user portraits; performing disease type prediction on the target user based on the anti-illusion prediction model, the full-modal fusion feature, and the user label, and performing weight distribution of each modality data based on the analysis result to obtain a weight distribution matrix; The full-modal fusion features and the weight distribution matrix are weighted and fused, and based on the fusion result and the disease type prediction result, the disease type of the target user and the corresponding probability thereof are predicted. 5.The attention mechanism based multi-modal data processing method of claim 1, wherein, The preprocessing of the original multi-modal array of the target user comprises: Based on the entity relationship knowledge graph, the KL divergence of each term in the original multi-modal array in different modal data is calculated, and each term is subjected to ambiguity detection based on the KL divergence; When an ambiguous term is detected in each term, the ambiguous term is subjected to semantic disambiguation based on the attention weight of each modal data in the entity relationship knowledge graph. 6.The attention mechanism based multi-modal data processing method of claim 1, wherein, Before determining the attention weight of each modal data in the original multi-modal array based on the preset entity relationship knowledge graph and the structured feature vector group, the method further comprises: The structured feature vector is subjected to feature decoding, CUI mapping and encoding fusion to obtain a standardized entity feature vector; Based on the entity relationship knowledge graph, the dynamic relationship between each entity in the standardized entity feature vector is determined; Based on the dynamic relationship, cross-modal correlation modeling is performed to obtain each entity relationship group, and the entity relationship knowledge graph is generated based on each entity relationship group.

7. The attention mechanism based multi-modal data processing method of any one of claims 1-6, wherein, Before predicting the disease type of the target user and the corresponding probability thereof based on the preset user label, the anti-illusion prediction model and the full-modal fusion features, the method further comprises: Based on the time delay mutual information causal discovery algorithm, the original medical multi-modal data is time-aligned and preprocessed, and the cross-modal causal relationship between each modal data is extracted from the aligned modal data to obtain a cross-modal causal graph. Based on the preset dynamic gating mechanism and the cross-modal causal graph, the weight of each modal data in the to-be-trained prediction model is adjusted to obtain the target weight corresponding to each modal data. The modal contradiction of the original medical multi-modal data is constructed to generate an adversarial multi-modal sample including a conflict label. Based on the target weight and the cross-modal causal graph, the to-be-trained model is subjected to causal regularization constraint adversarial training to obtain the anti-illusion prediction model.

8. An attention mechanism based multi-modal data processing apparatus, characterized in that, Comprise: The structural feature extraction module is configured to preprocess the original multi-modal array of the target user, and extract features from the preprocessed original multi-modal array to generate a structured feature vector group, wherein the original multi-modal array comprises image data, electronic medical records, genetic data and behavior data. The modal weight determination module is configured to determine the attention weight of each modal data in the original multi-modal array based on the preset entity relationship knowledge graph and the structured feature vector group. The modal feature fusion module is configured to perform hierarchical attention fusion on the structured feature vector based on the attention weight of each modal data to generate full-modal fusion features. The user disease prediction module is configured to predict the disease type of the target user and the corresponding probability thereof based on the preset user label, the anti-illusion prediction model and the full-modal fusion features.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The computer program is executed by the processor to implement the steps of the attention mechanism based multi-modal data processing method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 9. The computer program is executed by the processor to implement the steps of the attention mechanism based multi-modal data processing method according to any one of claims 1 to 7.

Images