Construction method of cross-modal time sequence diagram model of complex common disease network

Abstract

The invention relates to a method for constructing a cross-modal time sequence diagram model of a complex common disease network, and belongs to the technical field of common disease networks, and the method comprises the following steps: S1, dividing the diagnosis and treatment data of a patient into text data, image data and time sequence data classification; s2, processing text information by adopting a word embedding model, processing image information by adopting a convolutional neural network, and processing time sequence data by adopting a time sequence modeling module; then, a cross-modal multi-head attention CM-MHA module is adopted to carry out cross-modal fusion on the three features, and a feature matrix Ffuse with a time sequence is formed; s3, according to the current diagnosis and treatment information of the patient, establishing a complex common disease network by adopting an advantage ratio RR method; and S4, embedding the feature matrix Ffuse with the time sequence into the complex common disease network by adopting a graph embedding technology, and according to a time sequence data updating rule, forming a cross-modal time sequence common disease network.

Description

Technical Field

[0001] The present invention belongs to the technical field of comorbidity networks and relates to a method for constructing a cross-modal temporal graph model of a complex comorbidity network. Background Art

[0002] Current clinical practice primarily focuses on the treatment of single diseases, with limited understanding of the coexistence of multiple diseases. Comorbidity networks, as an emerging discipline, represent the complex relationships between diseases in a network format and describe them through topological metrics. This can enhance understanding of comorbidities and their relationships, providing new perspectives for understanding comorbidity mechanisms. Numerous researchers have applied comorbidity networks to various chronic disease areas. Yang Qingqing et al. collected diagnostic information from patients with chronic obstructive pulmonary disease (COPD) in a cross-sectional manner to establish a comorbidity network. They also used the Louvain method to explore comorbidity patterns in patients of different genders, providing a theoretical basis for stratified patient management. Aguado et al. used electronic health records (EHRs) from over three million adults in Catalonia, Spain, to construct a comorbidity network and extract subnetworks focused on common chronic diseases. Liu Chang et al. used data from the 2020 China Health and Retirement Longitudinal Study (CHARLS) to construct a comorbidity network for middle-aged and elderly patients with chronic diseases in my country. They also explored comorbidity patterns and networks, finding a high prevalence of comorbidity among middle-aged and elderly patients. In the field of cardiovascular disease, Li Yanna and others used the medical record information data of hospitalized patients with ischemic cardiovascular disease (IHD) to construct an IHD comorbidity network and analyzed the characteristics and patterns of comorbidities. Tang et al. also used the EHR data of 14.3 million patients to construct an undirected network of cardiovascular disease comorbidities, and designed a new method - progression rates (PR) to further analyze the progression relationship between diseases. However, it is worth noting that the above studies all used cross-sectional diagnosis and treatment data of patients at a certain time point to construct comorbidity networks, without considering the patient's disease course time series information and imaging and other multimodal features, making it difficult to accurately reflect the overall picture and dynamic changes of the connection between comorbidities. Therefore, the present invention will propose a new method to embed the patient's diagnosis and treatment time series information while incorporating multimodal information such as imaging and test data to construct a time series comorbidity network, laying the foundation for future disease course evolution reasoning and personalized treatment. Summary of the Invention

[0003] In view of this, the purpose of the present invention is to integrate patient imaging information and time series information to construct a cross-modal time series disease comorbidity network on the basis of the existing disease comorbidity network, and provide a method for constructing a cross-modal time series graph model of a complex comorbidity network.

[0004] In order to achieve the above object, the present invention provides the following technical solutions:

[0005] A method for constructing a cross-modal temporal graph model of a complex comorbidity network includes the following steps:

[0006] S1: Divide patient diagnosis and treatment data into text data, image data and time series data categories;

[0007] S2: Use word embedding model to process text information, convolutional neural network to process image information, and time series modeling module to process time series data; then use Cross-Modal Multi-Head Attention (CM-MHA) module to perform cross-modal fusion of the three features to form a feature matrix F with time series. fused ;

[0008] S3: Based on the patient's current diagnosis and treatment information, the odds ratio (RR) method was used to construct a complex comorbidity network;

[0009] S4: Use graph embedding technology to embed the time-series feature matrix F fused Embed complex comorbidity networks and update rules based on time series data to form a cross-modal time series comorbidity network.

[0010] Furthermore, the word embedding model is used to process text information in step S2, specifically including:

[0011] First, the text is preprocessed by word segmentation and tokenization. The preprocessed text sequence is {x1,x2...,x N}, use Word2Vec for word embedding:

[0012] L i =E[x i ]

[0013] Among them, L i represents the embedding vector, E is the word embedding matrix, x i It is a word index.

[0014] Furthermore, the convolutional neural network is used to process the image information in step S2, specifically including:

[0015] Split the input image I into several small blocks of size P×P and flatten each block into a one-dimensional vector of length P 2 ×C, where C is the number of image channels;

[0016] The block sequence after segmentation is in, H and W are the height and width of the input image respectively;

[0017] Each block passes through a linear projection layer from P 2 ×C is mapped to a fixed dimension d1, expressed as:

[0018] x′ i =Wpatch ·flatten(x i )+b patch

[0019] in is the weight matrix, is the bias term, flatten(x i ) means to convert x i Flattened into a one-dimensional vector;

[0020] Add a positional embedding to each chunk Adding the features of the block to preserve the image order gives:

[0021] x″ i =x′ i +E pos，i

[0022] Among them, E pos，i is the embedding of the i-th position;

[0023] x″ i Input into the Transformer encoder, and after passing through the L-layer encoder, the output is:

[0024] X out =TransformerEncoder({x″1,x″2,...,x″ N})

[0025] Finally, all features are concatenated into a long vector:

[0026] I cat =[X out,1 ;X out,2 ;...;X out,N ].

[0027] Furthermore, the time series modeling module used in step S2 to process time series data is to use a retrieval-enhanced learner (RAL) to extract high-level time features based on block-based processing and a memory-enhanced attention mechanism, including the following steps:

[0028] The time series Divide into overlapping blocks of length patch_len, where B is the batch size, L is the sequence length, and D is the number of variables;

[0029] Each block is mapped to a fixed dimension d2 through a linear projection layer and position embedding is added to preserve the temporal order. The resulting block embedding t″ i local temporal patterns are captured;

[0030] Memory-enhanced attention is used to capture the embedded temporal modules, using a set of learnable memory queries Q∈ Interact with block embeddings through multi-head attention; block embeddings are projected into K key vectors and V value vectors, and calculated:

[0031]

[0032] Get the time series vector set

[0033] Furthermore, in step S2, the cross-modal multi-head attention CM-MHA module is used to perform cross-modal fusion on the three features to form a feature matrix F with time series. fused , specifically including:

[0034] First, the generated images and texts are processed by the visual language model ViLT to form Multimodal embedding of, where B is the batch size, L f is the sequence length, d n is the hidden layer dimension; the time series vector set T obtained after reinforcement learning is used as the query vector, F m As key vector and value vector, define cross-modal multi-head attention CM-MHA as follows:

[0035] CM-MHA(Q,K,v)=Cat(head1,head2,...,head n )W O

[0036]

[0037] Where Q = TW Q , K=F m W K , B=F m / w V , and W O is the learnable projection matrix; d k = d2 / n is the dimension of each attention head, n is the number of attention heads; Q represents the query vector in cross-modal multi-head attention, K and V represent the key and value in cross-modal multi-head attention, respectively;

[0038] The training process is stabilized by residual and normalization to obtain F attn :

[0039] F attn =LayerNorm(T+CM-MHA(Q,K,V))

[0040] Finally, a gating mechanism is used to adaptively balance temporal features and multimodal features, thereby achieving robust fusion:

[0041] G=σ(Wg [T; F m ]+b g )

[0042] F fused =G⊙F attn +(1-G)⊙F m

[0043] Among them, W g and b g is the learning parameter, σ is the activation function, F m Represents the multimodal vector after fusing text and image, F attn Represents the cross-modal vector after fusion temporal stabilization training.

[0044] Furthermore, in step S3, nodes are used to represent diseases, edges are used to represent disease relationships, and a time decay factor is introduced to improve the odds ratio (RR):

[0045]

[0046] Among them, RR ij (t) represents the association strength between disease i and disease j at time t; I ik (t) and I jk (t) represents the indicator function of whether the kth patient has disease i and disease j at time t, 1 means having disease i and 0 means not having disease j; E i (t) and E j (t) represents the expected prevalence of disease i and disease j in the population; α is the time decay coefficient, which controls the decay rate of the time effect; d(t,t k ) represents the current time t and the diagnosis and treatment time t of patient k k The interval between them; N is the total number of patients.

[0047] Further, step S4 specifically includes the following steps:

[0048] As the comorbidity network evolves over time, it updates the adjacency matrix of the network:

[0049] A(t+1)=σ(W·[F fused (t)||F fused (t-1)]+b)

[0050] Where A(t+1) represents the adjacency matrix of the comorbidity network at time t+1; σ is the Sigmoid activation function, which is used to map the output value to the interval [0,1], indicating the probability of association between diseases; W is the learnable weight matrix; b is the bias term; F fused (t) and F fused(t-1) represents the fused multimodal time series feature matrix at time t and t-1, respectively; || represents the feature concatenation operation. The beneficial effect of the present invention lies in: building on existing comorbidity networks, the present invention integrates cross-modal patient diagnosis and treatment information and time series, enabling the construction of a cross-modal dynamic disease network. This technical application can provide decision-making references for downstream tasks such as patient disease course deduction, personalized diagnosis and treatment, prognosis plans, and outcome prediction.

[0051] Other advantages, objects, and features of the present invention will be described in part in the following description and, in part, will be apparent to those skilled in the art upon examination of the following description or may be learned from practice of the present invention. The objects and other advantages of the present invention may be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below with reference to the accompanying drawings, in which:

[0053] Figure 1 Constructing a flow chart for a cross-modal temporal comorbidity network. DETAILED DESCRIPTION

[0054] The following describes the embodiments of the present invention by means of specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0055] It should be noted that the illustrations provided in the following embodiments are merely schematic illustrations of the basic concept of the present invention. Therefore, the illustrations only show components related to the present invention and are not drawn according to the number, shape, and size of components in actual implementation. In actual implementation, the type, quantity, and proportion of each component may be changed arbitrarily, and the component layout may also be more complex.

[0056] In the following description, numerous details are discussed to provide a more thorough explanation of the embodiments of the present invention. However, it will be apparent to those skilled in the art that the embodiments of the present invention may be practiced without these specific details. In other embodiments, well-known structures and devices are shown in block diagram form rather than in detail to avoid obscuring the embodiments of the present invention.

[0057] Example 1:

[0058] like Figure 1 As shown, the present invention provides a method for constructing a cross-modal temporal graph model of a complex comorbidity network, comprising the following steps:

[0059] The first step is to collect and organize the personal diagnosis and treatment data of cardiovascular disease patients, and classify text data (such as XML or natural language format), imaging data (including CT, MRI and other images), and time series data (such as electrocardiogram, blood pressure monitoring and other test results containing continuous time series);

[0060] Step 2: Use word embedding models (such as Word2Vec, BERT, etc.), convolutional neural networks (such as ViT, ResNet, etc.) and time series modeling modules (such as RAL, LSTM, etc.) to process text information, image information and time series data. Then use the cross-modal multi-head attention (CM-MHA) module to perform cross-modal fusion of the three features to form a feature matrix F with time series. fused ;

[0061] Step 3: Based on the patient's current diagnosis and treatment information, the odds ratio (RR) method was used to construct a cardiovascular disease comorbidity network;

[0062] Step 4: Use graph embedding technology to embed the time-series feature matrix F fused The cardiovascular disease comorbidity network is embedded and the rules are updated according to the temporal data (e.g., every 15-30 days) to form a cross-modal temporal comorbidity network.

[0063] Example 2:

[0064] In this embodiment, a specific construction example of a method for constructing a cross-modal temporal graph model of a complex comorbidity network is provided. First, the patient's electronic medical record data is divided into text information (including medical records, diagnosis results, etc.), imaging data (including ultrasound, CT, MRI, etc.) and time series data (including physiological indicator treatment records, drug use records, etc.). Word2Vec, Vision Transformer (ViT) and Retrieval-Augmented Learner (RAL) are used to process text information, imaging information and time series data respectively. Then, a cross-modal multi-head attention mechanism (CM-MHA) and a gated fusion mechanism are used to realize multimodal fusion and embed the patient's comorbidity network to realize the construction of a multimodal temporal comorbidity network. This method specifically includes the following steps:

[0065] (1) Text information processing:

[0066] First, the text is preprocessed by word segmentation and tokenization. The preprocessed text sequence is {x1,x2...,x N}, use Word2Vec for word embedding:

[0067] L i =E[x i ]

[0068] Among them, L i represents the embedding vector, E is the word embedding matrix, x i It is a word index.

[0069] (2) Image information processing:

[0070] Split the input image I into several small blocks of size P×P and flatten each block into a one-dimensional vector of length P 2 ×C, where C is the number of image channels. The block sequence after segmentation is in, H and W are the height and width of the input image respectively. Each block is passed through a linear projection layer from P 2 ×C is mapped to a fixed dimension d1, and the linear projection layer can be expressed as:

[0071] x′ i =W patch ·flatten(x i )+b patch

[0072] in, is the weight matrix, is the bias term, flatten(x i ) means to convert x i Flattened to a one-dimensional vector.

[0073] Add a position embedding to each block Adding the features of the block to preserve the image order gives:

[0074] x′ i '=x′ i +E pos，i

[0075] Among them, E pos，i is the embedding of the ith position.

[0076] x″ i Input into the Transformer encoder, and after passing through the L-layer encoder, the output is:

[0077] X out =TransformerEncoder({x″1,x″2,...,x″N})

[0078] Finally, all features are concatenated into a long vector:

[0079] I cat =[X out,1 ;X out,2 ;...;X out,N ]

[0080] (3) Timing information processing:

[0081] The Retrieval Enhanced Learner (RAL) module extracts high-level temporal features through block-based processing and memory-enhanced attention mechanism. It mainly consists of two steps. First, the time series Divide into overlapping blocks of length patch_len, where B is the batch size, L is the sequence length, and D is the variable. Similar to image information, each block is mapped to a fixed dimension d2 through a linear projection layer, and position embedding is added to preserve the temporal order. The resulting block embedding t″ i Local temporal patterns are captured.

[0082] Then, memory-enhanced attention is used to capture the embedded temporal modules, using a set of learnable memory queries. Interact with block embeddings through multi-head attention. Block embeddings are projected into K key vectors and V value vectors and calculated:

[0083]

[0084] Get the time series vector set Cross-modal dynamic retrieval and fusion can be performed.

[0085] (4) Cross-modal feature fusion:

[0086] First, the generated images and texts are processed by the Vision-and-Language Transformer (ViLT) to form Where B is the batch size, L f is the sequence length, d n is the hidden layer dimension. The time series vector set T obtained after reinforcement learning is used as the query vector, F m As the key vector and value vector, define the cross-modal multi-head attention (CM-MHA) as follows:

[0087] CM-MHA(Q,K,V)=Cat(head1,head2,...,head n )W O

[0088]

[0089] Where Q represents the query vector in cross-modal multi-head attention, Q = TW Q ; K and V represent the key and value in cross-modal multi-head attention, respectively, K = F m W K , V=F m W V ;W i Q , W i K , W i V and W O is the learnable projection matrix; d k =d2 / n is the dimension of each attention head, and n is the number of attention heads. This mechanism aligns and integrates temporal features and multimodal features, which can capture fine-grained and high-level contextual information.

[0090] The training process is stabilized by residual and normalization to obtain F attn :

[0091] F attn =LayerNorm(T+CM-MHA(Q,K,V))

[0092] Finally, a gating mechanism is used to adaptively balance temporal features and multimodal features, thereby achieving robust fusion:

[0093] G=σ(W g [T; F m ]+b g )

[0094] F fused =G⊙F attn +(1-G)⊙F m

[0095] Among them, W g and b g is the learning parameter, σ is the activation function, F m Represents the multimodal vector after fusing text and image, F attn Represents the cross-modal vector after fusion temporal stabilization training.

[0096] (5) Construction of a multimodal temporal comorbidity network

[0097] Nodes represent diseases, and edges represent disease relationships between diseases. When constructing the network, in order to reflect the dynamic change of disease association strength over time, the present invention introduces a time decay factor to improve the traditional odds ratio (RR):

[0098]

[0099] Among them, RR ij (t) represents the association strength between disease i and disease j at time t; I ik (t) and I jk (t) are indicator functions indicating whether the kth patient has disease i or disease j at time t (1 means having disease, 0 means not having disease); E i (t) and E j (t) represents the expected prevalence of disease i and disease j in the population; α is the time decay coefficient, which controls the decay rate of the time effect; d(t,t k ) represents the current time t and the diagnosis and treatment time t of patient k k The interval between them; N is the total number of patients.

[0100] The time decay factor reflects the timeliness of disease association. Recent diagnosis and treatment data have a greater impact on the intensity of disease association, while the impact of older data gradually weakens, which is more in line with the dynamic change law of disease association in actual clinical practice. When the comorbidity network evolves over time, the adjacency matrix of the updated network is:

[0101] A(t+1)=σ(W·[F fused (t)||F fused (t-1)]+b)

[0102] Where A(t+1) represents the adjacency matrix of the comorbidity network at time t+1; σ is the Sigmoid activation function, which is used to map the output value to the interval [0,1], indicating the probability of association between diseases; W is the learnable weight matrix; b is the bias term; F fused (t) and F fused (t-1) represents the multimodal temporal feature matrix after fusion at time t and t-1 respectively; || represents the feature concatenation operation.

[0103] In the above embodiments, references to "this embodiment" in the specification indicate that a particular feature, structure, or characteristic described in conjunction with the embodiment is included in at least some embodiments, but not necessarily all embodiments. Multiple occurrences of "this embodiment" do not necessarily refer to the same embodiment.

[0104] In the above embodiments, although the invention has been described in conjunction with specific embodiments thereof, many alternatives, modifications, and variations of these embodiments will be apparent to those skilled in the art based on the foregoing description. For example, other memory structures (e.g., dynamic RAM (DRAM)) may be used with the embodiments discussed. The embodiments of the present invention are intended to encompass all such alternatives, modifications, and variations that fall within the broad scope of the appended claims.

[0105] This embodiment further provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, any one of the methods in this embodiment is implemented.

[0106] This embodiment also provides an electronic terminal, including: a processor and a memory;

[0107] The memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory, so that the terminal executes any one of the methods in this embodiment.

[0108] Regarding the computer-readable storage medium in this embodiment, those skilled in the art will appreciate that all or part of the steps in the aforementioned method embodiments can be implemented using hardware associated with the computer program. The aforementioned computer program can be stored in a computer-readable storage medium. When executed, the program performs the steps in the aforementioned method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0109] The electronic terminal provided in this embodiment includes a processor, a memory, a transceiver and a communication interface. The memory and the communication interface are connected to the processor and the transceiver and complete communication with each other. The memory is used to store computer programs, the communication interface is used for communication, and the processor and the transceiver are used to run computer programs so that the electronic terminal executes the various steps of the above method.

[0110] In this embodiment, the memory may include a random access memory (RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk storage.

[0111] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, and discrete hardware components.

[0112] The present invention can be used in a wide variety of general-purpose or special-purpose computing system environments or configurations, such as personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments that include any of the above.

[0113] The present invention may be described in the general context of computer-executable instructions, such as program modules, executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. The present invention may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media, including storage devices.

[0114] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not limiting. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solutions, which should all be included in the scope of the claims of the present invention.

Claims

1. A method for constructing a cross-modal temporal graph model of a complex comorbidity network, characterized by: The following steps are involved: S1: Divide patient diagnosis and treatment data into text data, image data and time series data categories; S2: Use word embedding model to process text information, convolutional neural network to process image information, and time series modeling module to process time series data; then use cross-modal multi-head attention CM-MHA module to cross-modally fuse the three features to form a feature matrix F with time series. fused ; S3: Based on the patient's current diagnosis and treatment information, the odds ratio (RR) method was used to construct a complex comorbidity network; S4: Use graph embedding technology to embed the time-series feature matrix F fused Embed complex comorbidity networks and update rules based on time series data to form a cross-modal time series comorbidity network.

2. The method for constructing a cross-modal temporal graph model of a complex comorbidity network according to claim 1, characterized in that: Step S2 uses a word embedding model to process text information, specifically including: First, the text is preprocessed by word segmentation and tokenization. The preprocessed text sequence is {x1,x2...,x N }, use Word2Vec for word embedding: THE i =E[x i ] Among them, L i represents the embedding vector, E is the word embedding matrix, and xi is the word index.

3. The method for constructing a cross-modal temporal graph model of a complex comorbidity network according to claim 1, characterized in that: Step S2 uses a convolutional neural network to process image information, specifically including: Split the input image I into several small blocks of size P×P and flatten each block into a one-dimensional vector of length P 2 ×C, where C is the number of image channels; The block sequence after segmentation is in, H and W are the height and width of the input image respectively; Each block passes through a linear projection layer from P 2 ×C is mapped to a fixed dimension d1, expressed as: x′ i =W patch ·flatten(x i )+b patch in is the weight matrix, is the bias term, flatten(x i ) means to convert x i Flattened into a one-dimensional vector; Add a positional embedding to each chunk Adding the features of the block to preserve the image order gives: x″ i =x′ i +E pos，i Among them, E pos，i is the embedding of the i-th position; x″ i Input into the Transformer encoder, and after passing through the L-layer encoder, the output is: X out =TransformerEncoder({x″1,x″2,...,x″ N }) Finally, all features are concatenated into a long vector: I cat =[X out,1 ;X out,2 ;...;X out,N ]。 4. The method for constructing a cross-modal temporal graph model of a complex comorbidity network according to claim 1, characterized in that: The time series modeling module used in step S2 to process time series data uses a retrieval-enhanced learner (RAL) to extract high-level temporal features based on block-based processing and a memory-enhanced attention mechanism, including the following steps: The time series Divide into overlapping blocks of length patch_len, where B is the batch size, L is the sequence length, and D is the number of variables; Each block is mapped to a fixed dimension d2 through a linear projection layer and position embedding is added to preserve the temporal order. The resulting block embedding t″ i local temporal patterns are captured; Memory-enhanced attention is used to capture the embedded temporal modules, using a set of learnable memory queries Interact with block embeddings through multi-head attention; block embeddings are projected into K key vectors and V value vectors, and calculated: Get the time series vector set 5. The method for constructing a cross-modal temporal graph model of a complex comorbidity network according to claim 4, characterized in that: In step S2, the cross-modal multi-head attention CM-MHA module is used to cross-modally fuse the three features to form a feature matrix F with time series. fused , specifically including: First, the generated images and texts are processed by the visual language model ViLT to form Multimodal embedding of, where B is the batch size, L f is the sequence length, d n is the hidden layer dimension; the time series vector set T obtained after reinforcement learning is used as the query vector, F m As key vector and value vector, define cross-modal multi-head attention CM-MHA as follows: CM-MHA(Q,K,V)=Cat(head1,head2,...,head n )W O Where Q = TW Q , K=F m W K , V=F m W V , W i Q , W i K , W i V and W O is the learnable projection matrix; d k = d2 / n is the dimension of each attention head, n is the number of attention heads; Q represents the query vector in cross-modal multi-head attention, K and V represent the key and value in cross-modal multi-head attention, respectively; The training process is stabilized by residual and normalization to obtain F attn : F attn =LayerNorm(T+CM-MHA(Q,K,V)) Finally, a gating mechanism is used to adaptively balance temporal features and multimodal features, thereby achieving robust fusion: G=σ(W g [T;F m ]+b g ) F fused =G⊙F attn +(1-G)⊙F m Among them, W g and b g is the learning parameter, σ is the activation function, F m Represents the multimodal vector after fusing text and image, F attn Represents the cross-modal vector after fusion temporal stabilization training.

6. The method for constructing a cross-modal temporal graph model of a complex comorbidity network according to claim 1, characterized in that: In step S3, nodes are used to represent diseases, edges are used to represent disease relationships, and a time decay factor is introduced to improve the odds ratio (RR): Among them, RR ij (t) represents the association strength between disease i and disease j at time t; I ik (t) and I jk (t) represents the indicator function of whether the kth patient has disease i and disease j at time t, 1 means having disease i and 0 means not having disease j; E i (t) and E j (t) represents the expected prevalence of disease i and disease j in the population; α is the time decay coefficient, which controls the decay rate of the time effect; d(t,t k ) represents the current time t and the diagnosis and treatment time t of patient k k The interval between them; N is the total number of patients.

7. The method for constructing a cross-modal temporal graph model of a complex comorbidity network according to claim 1, characterized in that: Step S4 specifically includes the following steps: As the comorbidity network evolves over time, it updates the adjacency matrix of the network: A(t+1)=σ(W·[F fused (t)||F fused (t-1)]+b) Where A(t+1) represents the adjacency matrix of the comorbidity network at time t+1; σ is the Sigmoid activation function, which is used to map the output value to the interval [0,1], indicating the probability of association between diseases; W is the learnable weight matrix; b is the bias term; F fused (t) and F fused (t-1) represents the multimodal temporal feature matrix after fusion at time t and t-1 respectively; || represents the feature concatenation operation.

Images