Multi-modal traditional chinese medicine feature fusion method and system based on dynamic tensor alignment

Abstract

The application discloses a kind of multi-modal traditional Chinese medicine feature depth fusion method and system based on dynamic tensor alignment mechanism, belong to traditional Chinese medicine information processing and artificial intelligence cross technical field.The technical principle of its invention is as follows:First, the regular tensor of drug mode and the nested tensor of component mode are constructed, and the structure correlation is maintained by alignment index;Then the variable-length component sequence is encoded into a fixed-dimensional aggregated tensor through an adaptive aggregation module;Then, a bidirectional cross-attention mechanism incorporating compatibility prior is used to realize the deep interaction of drug and component information;Finally, a gradual fusion strategy is used to generate a fusion feature tensor.This method solves the technical problems of existing technology, such as the inability to maintain the hierarchical structure of drugs and chemical components, the low efficiency of processing variable-length sequences, and the neglect of traditional Chinese medicine compatibility prior knowledge.This method has the beneficial effects of maintaining data structure, high computational efficiency, realizing bidirectional information flow, incorporating domain knowledge, and stable and controllable fusion process.

Description

Technical Field

[0001] This invention belongs to the field of traditional Chinese medicine information processing and artificial intelligence technology, specifically a method and system for multimodal traditional Chinese medicine feature fusion based on dynamic tensor alignment. Background Technology

[0002] Traditional Chinese medicine (TCM) formulas are the core carriers of TCM clinical treatment, and their modernization research relies on the integrated analysis of multi-source heterogeneous data. On the one hand, formulas contain rich drug-level information, including macroscopic medical knowledge such as the names, properties, meridian tropism, dosages, and compatibility relationships of each herb; on the other hand, each herb itself contains dozens to hundreds of chemical components with complex molecular structures, pharmacological activities, and microscopic mechanisms of action. Therefore, constructing a mapping relationship from microscopic chemical components to macroscopic formula efficacy, and achieving the effective integration of drug modal and chemical component modal information, is a key issue in promoting the intelligent analysis and innovation of TCM formulas.

[0003] Currently, in the field of multimodal data processing, existing fusion methods mainly include early fusion, late fusion, and attention-based fusion. Early fusion methods simply concatenate the feature vectors of different modalities before inputting them into the model. This method ignores the complex interactions between modalities and cannot distinguish the importance of different features. Late fusion methods first model each modality independently and then integrate the results only at the decision layer. While preserving modal characteristics, this completely ignores the potential complementary and reinforcing relationships between modalities. Attention-based fusion methods can dynamically allocate weights, but they typically assume that features from different modalities have the same sequence length and semantic granularity. In the field of traditional Chinese medicine, this assumption does not hold: the number of chemical components contained in different drugs varies greatly, ranging from a few to hundreds, making it difficult to directly apply traditional attention mechanisms.

[0004] A more prominent problem is that existing fusion technologies generally treat traditional Chinese medicine (TCM) data as a flat structure, severely neglecting the crucial prior structure of the inherent hierarchical attribution relationship between "formula-drug-chemical components." In TCM formulas, each chemical component clearly belongs to a specific drug, and this hierarchical structure contains important semantic information. For example, different components originating from the same drug often have synergistic effects, while components originating from different drugs may produce compatibility effects such as "mutual reinforcement, mutual assistance, mutual restraint, and mutual antagonism." Existing methods treat all chemical components as an undifferentiated long sequence, completely losing the attribution relationship between drugs and components. This results in the model being unable to distinguish the origin of components, making it difficult to accurately model the complex interactions within the formula, and severely limiting the discriminative power and interpretability of fusion features.

[0005] Furthermore, regarding computational efficiency, existing methods typically employ a fixed-length padding strategy to handle variable-length chemical component sequences, uniformly padding the sequences to the maximum length before batch computation. In the context of traditional Chinese medicine with vastly different numbers of chemical components, this approach introduces a large amount of invalid padding computation, significantly increasing computational complexity and memory overhead. Additionally, noise at the padding locations can interfere with model training.

[0006] In summary, existing multimodal fusion technologies have the following main drawbacks when applied to the field of traditional Chinese medicine (TCM): first, they cannot maintain and utilize the hierarchical structural information between drugs and chemical components; second, they struggle to effectively handle variable-length sequences with extremely uneven distributions of chemical components; third, they suffer from low computational efficiency and significant redundancy; and fourth, they fail to organically integrate TCM knowledge (such as compatibility theory) into data-driven models. Therefore, there is an urgent need for a novel multimodal feature deep fusion method that can address these issues to better support intelligent analysis tasks such as TCM prescription efficacy prediction, new drug discovery, and compatibility rule mining. Summary of the Invention

[0007] To address the above problems, this invention provides a multimodal TCM feature fusion method and system based on dynamic tensor alignment, which solves the problems of not being able to maintain and utilize the hierarchical structural information between drugs and chemical components, the significant differences in the number of different drug components, and low computational efficiency.

[0008] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0009] A deep fusion method for multimodal TCM features based on dynamic tensor alignment mechanism includes the following steps:

[0010] S100: Obtain the list of drug names and chemical component identifiers of traditional Chinese medicine prescriptions, perform embedded vector queries of drugs and components, and construct an aligned index structure;

[0011] S200. Based on drug embedding vectors and component embedding vectors, construct a two-dimensional tensor representation of drug modalities and a nested tensor representation of component modalities.

[0012] S300. Perform adaptive aggregation processing on the nested tensor representation of the component modes to generate a component aggregation tensor;

[0013] S400: Based on the preset Chinese herbal medicine compatibility knowledge base and the drug list of the input prescription, calculate the compatibility prior bias matrix;

[0014] S500, based on drug modality two-dimensional tensor representation, component aggregation tensor and compatibility prior bias matrix, performs bidirectional cross-attention calculation to generate bidirectional interactive features;

[0015] S600, based on bidirectional interaction features, two-dimensional tensor representation of drug modality and component aggregation tensor, performs progressive fusion processing to generate fused feature tensor;

[0016] S700 outputs the fused feature tensor to the downstream task model.

[0017] The beneficial effects of this technical solution are as follows: it fully preserves the hierarchical relationship between drugs and chemical components, improving the semantic accuracy and interpretability of feature fusion; it avoids invalid filling calculations caused by significant differences in the number of components by using nested tensors and adaptive aggregation, significantly improving computational efficiency and memory utilization; it achieves deep interaction and complementary enhancement of drug and component information through bidirectional cross-attention; it integrates prior knowledge of TCM compatibility into the model, enhancing the model's generalization ability and domain adaptability; and it adopts a progressive fusion strategy, making the fusion process more stable and controllable, effectively preserving key information from each modality.

[0018] Furthermore, the construction of the alignment index structure in step S100 specifically includes: querying the thesaurus based on the drug name list to obtain the corresponding drug identifier sequence; querying the drug-component mapping table based on the drug identifier sequence to obtain the chemical component identifier list corresponding to each drug; and calculating the start index and end index of the chemical component corresponding to each drug in the flattened sequence based on the length of each chemical component identifier list to form an alignment index set.

[0019] The beneficial effects of this technical solution are as follows: by using two-level queries (drug name → drug ID → ingredient ID list) and length-based index calculation, unstructured list data is transformed into structured index mapping relationships, providing accurate drug-ingredient correspondences and ensuring the correctness of information attribution during subsequent tensor construction and fusion processes.

[0020] Furthermore, step S200 specifically includes: stacking the drug embedding vectors in drug order to form a two-dimensional tensor. Where n is the number of drugs and d is the embedding dimension; the embedding vectors of the chemical components of each drug are organized into matrices to form a nested tensor set. ,in This represents the number of chemical components in the i-th drug.

[0021] The beneficial effects of this technical solution are as follows: the drug tensor provides a regular representation of the prescription at the macro level, while the nested component tensor flexibly accommodates a varying number of micro-component information without destroying the attribution relationship.

[0022] Furthermore, the adaptive aggregation processing of the nested tensor representation of the component modes in step S300 includes enhancement via a gated residual mechanism. The enhancement via the gated residual mechanism specifically includes:

[0023] Obtain the aggregated token representation output by the component-level Transformer encoder. ;

[0024] Calculate the gating factor ;

[0025] Computational Enhanced Aggregate Representation ,in, This is the initial aggregation representation. For the enhanced aggregation representation, , , , For learnable parameters, This represents element-wise multiplication; the enhanced polymerization representations of all drugs are stacked into a component polymerization tensor. .

[0026] The beneficial effects of this technical solution are as follows: by capturing the internal relationships between components through the Transformer encoder, and then dynamically determining the degree of nonlinear enhancement to be applied to the aggregation result through a learnable nonlinear gating mechanism, the aggregation representation can more accurately summarize the core component characteristics of the drug and avoid information loss caused by simple pooling.

[0027] Furthermore, the specific process of performing bidirectional cross-attention calculation in step S500 includes: processing the drug tensor... and component polymerization tensor Perform linear projections to obtain the query vector, key vector, and value vector; calculate the attention score from drug to component. ,in This is a drug query vector. For component key vectors, This is the transpose operation of a matrix. Let RPE be the scaling factor, RPE be the relative position encoding matrix, and Prior be the matching prior bias matrix; calculate the drug-to-component cross-attention output tensor based on the attention score. ,in Given a vector of component values; calculate the attention score from component to drug: ,in For component query vectors, For drug bond vectors, This is the transpose operation of a matrix. The scaling factor is used to calculate the component-to-drug cross-attention output tensor based on the attention score. ,in This is a vector of drug values.

[0028] The beneficial effects of this technical solution are as follows: it enables bidirectional query and reference of drug information and component information, allowing drug representation to be refined according to its component composition, and component representation to be adjusted according to the drug compatibility context; the introduced compatibility prior matrix makes the attention weight allocation conform to the principles of traditional Chinese medicine, enhancing the rationality and interpretability of the model.

[0029] Furthermore: Step S600, which sequentially performs element-level fusion, semantic-level fusion, and residual refinement, specifically includes the following: The progressive fusion module's process of performing element-level fusion, semantic-level fusion, and residual refinement specifically includes:

[0030] Element-level fusion: Calculate the output of element-level fusion. ,in , For learnable parameters, For the drug-to-component cross-attention output tensor, For the cross-attention output tensor from component to drug;

[0031] Semantic-level fusion: , and The concatenated feature tensor is obtained by concatenating the features. And compute the output tensor of semantic-level fusion. ,in , These are learnable parameters;

[0032] Residual Refinement: Computing the final fused feature tensor ,in LayerNorm is a learnable fusion strength parameter and a layer normalization operation.

[0033] The beneficial effects of this technical solution are as follows: preliminary modulation of information is achieved through gated element-level fusion; high-level feature integration is achieved through semantic-level fusion of splicing and nonlinear transformation; and finally, stable and controllable refined output is achieved through residual connections based on the original drug representation and learnable intensity parameters, effectively avoiding information loss or distortion during the fusion process.

[0034] Furthermore, step S400 specifically includes:

[0035] Based on the list of herbs in the input prescription, query the preset compatibility rating table to obtain any two herbs. and Compatibility score The scoring table is based on the compatibility relationships defined in classic Chinese medicine texts; the elements in the prior bias matrix are calculated. ,in For temperature coefficient, , It is a drug.

[0036] The beneficial effects of this technical solution are as follows: Qualitative compatibility relationships (such as mutual reinforcement, mutual enhancement, and mutual antagonism) are pre-quantified into specific numerical scores, constructing a static knowledge base; during model operation, based on the drug combination of the current prescription, the corresponding scores are dynamically extracted to form a prior matrix, which serves as the bias term of the attention mechanism. This provides the model with valuable domain prior knowledge, especially when the amount of data is limited, guiding the model to learn reasonable drug interaction patterns more quickly, improving the model's performance and generalization ability, while also giving the model's decision-making process a theoretical basis in traditional Chinese medicine.

[0037] Furthermore, the process of generating aggregation markers through the component-level Transformer encoder in step S300 specifically includes: adding a special aggregation marker [AGG] to the front of the chemical component sequence of each drug to form an extended sequence; inputting the extended sequence into the component-level Transformer encoder, which includes a multi-head self-attention mechanism and a feedforward neural network; and taking the hidden state at the position corresponding to the aggregation marker [AGG] in the sequence output by the encoder as the initial aggregation representation of the drug.

[0038] The beneficial effects of this technical solution are as follows: The aggregated token is used as a learnable query vector, interacting with all tokens (chemical components) in the sequence through a self-attention mechanism, and undergoing nonlinear transformation in a multi-layer feedforward network. Ultimately, its hidden state naturally contains global information converged from the entire sequence. Leveraging the powerful sequence modeling capabilities and self-attention mechanism of the Transformer, the aggregated token can fully "pay attention" to and fuse information from all chemical components in the sequence, thereby obtaining a context-aware vector representation that can represent the entire component set.

[0039] Furthermore, in step S400, the compatibility scoring table is based on the definition of compatibility relationships in classic Chinese medicine texts, and its construction rules are as follows:

[0040] If the two herbs are mutually reinforcing, then If the two herbs have a synergistic relationship, then If the two herbs are incompatible, then If the two herbs are in a toxic relationship, then If the two drugs are incompatible, then If the two herbs are incompatible, then If there is no clear compatibility, then .

[0041] The beneficial effects of this technical solution are as follows: Based on the inductive summary of classical TCM theories, each type of compatibility relationship is assigned a scalar value with direction (positive or negative) and intensity (absolute value). This value directly reflects the degree to which the relationship promotes or inhibits information interaction, providing a clear and quantitative prior knowledge injection scheme. This enables the model to clearly distinguish drug interactions of different natures (synergistic, inhibitory, antagonistic, etc.) and directly apply this distinction to the attention weight in numerical form, greatly enhancing the depth of integration between the model and domain theory.

[0042] Furthermore, a multimodal deep fusion system for traditional Chinese medicine features based on a dynamic tensor alignment mechanism, applied to any of the methods described above, includes: a data processing module for receiving input data, obtaining drug embedding vectors and component embedding vectors, and constructing an alignment index; a tensor construction module for constructing a two-dimensional tensor representation of the drug modality and a nested tensor representation of the component modality; an adaptive aggregation module for adaptively aggregating the nested tensor representation of the component modality to generate a component aggregation tensor; a priori calculation module for calculating the compatibility prior bias matrix; a bidirectional cross-attention module for performing bidirectional cross-attention calculation between the drug and the component; and a progressive fusion module. The module is used for progressively fusing bidirectional interaction features; the output module is used to output the final fused feature tensor; wherein, the output of the data processing module is connected to the tensor construction module; the output of the tensor construction module is connected to the adaptive aggregation module, the progressive fusion module and the bidirectional cross-attention module respectively; the output of the adaptive aggregation module is connected to the bidirectional cross-attention module and the progressive fusion module; the output of the prior calculation module is connected to the bidirectional cross-attention module; the output of the bidirectional cross-attention module is connected to the progressive fusion module; the output of the progressive fusion module is connected to the output module.

[0043] The beneficial effects of this technical solution are as follows: The system is designed according to the logical flow of data processing → tensor construction → feature aggregation → knowledge injection → cross-modal interaction → multi-level fusion → result output. The modules collaborate through clear interfaces (data format and connection relationship) to jointly complete the calculation task from raw Chinese medicine data to deeply fused features. Attached Figure Description

[0044] Figure 1 The flowchart shows a method for deep fusion of multimodal Chinese medicine features based on dynamic tensor alignment mechanism.

[0045] Figure 2 This is a schematic diagram of dynamic tensor alignment.

[0046] Figure 3 A heatmap showing the bidirectional cross-attention weights for the two drugs;

[0047] Figure 4 Feature distribution maps before and after fusion;

[0048] Figure 5 This is a block diagram of a multimodal deep fusion system for traditional Chinese medicine features based on a dynamic tensor alignment mechanism. Detailed Implementation

[0049] To enable those skilled in the art to better understand the technical solution, the present invention will be described in detail below with reference to embodiments. The description in this part is only exemplary and explanatory, and should not be used to limit the scope of protection of the present invention in any way.

[0050] Example 1

[0051] See attached document Figure 1 To be continued Figure 4 This embodiment provides a multimodal deep fusion method for traditional Chinese medicine features based on a dynamic tensor alignment mechanism, including: receiving input data, wherein the input data includes a list of drug names and a list of corresponding chemical components of traditional Chinese medicine prescriptions;

[0052] The data processing module obtains drug embedding vectors and component embedding vectors based on the drug name list and chemical component list, and constructs an alignment index. The alignment index records the start and end positions of the chemical components corresponding to each drug in the flattened sequence.

[0053] The tensor construction module is connected to the data processing module, receives the drug embedding vector and the component embedding vector, and constructs a two-dimensional tensor representation of the drug modality and a nested tensor representation of the component modality.

[0054] The adaptive aggregation module is connected to the tensor construction module, receives the nested tensor representation of the component modes, generates aggregation tags for the chemical component sequence of each drug through a component-level Transformer encoder, and enhances them through a gated residual mechanism to output the component aggregation tensor.

[0055] The prior calculation module calculates the compatibility score between each drug pair in the prescription based on a preset Chinese medicine compatibility knowledge base, and generates a compatibility prior bias matrix.

[0056] The bidirectional cross-attention module connects the adaptive aggregation module, the tensor construction module, and the prior calculation module. It receives the two-dimensional tensor representation of the drug modality, the component aggregation tensor, and the compatibility prior bias matrix. It performs cross-attention calculation from drug to component and cross-attention calculation from component to drug, and outputs bidirectional interactive features.

[0057] The progressive fusion module connects the bidirectional cross-attention module, the tensor construction module, and the adaptive aggregation module. It receives the bidirectional interactive features, the two-dimensional tensor representation of the drug modality, and the component aggregation tensor, and sequentially performs element-level fusion, semantic-level fusion, and residual refinement processing to output the fused feature tensor.

[0058] The output module is connected to the progressive fusion module and outputs the fused feature tensor to the downstream task model.

[0059] That is, the process includes at least steps S100 to S700:

[0060] S100: Obtain the list of drug names and chemical component identifiers of traditional Chinese medicine prescriptions, perform embedded vector queries of drugs and components, and construct an aligned index structure.

[0061] S200. Based on drug embedding vectors and component embedding vectors, construct a two-dimensional tensor representation of drug modalities and a nested tensor representation of component modalities.

[0062] S300. Perform adaptive aggregation processing on the nested tensor representation of the component modes to generate a component aggregation tensor;

[0063] S400: Based on the preset Chinese herbal medicine compatibility knowledge base and the drug list of the input prescription, calculate the compatibility prior bias matrix;

[0064] S500, based on the two-dimensional tensor representation of drug modality, component aggregation tensor and compatibility prior bias matrix, performs bidirectional cross-attention calculation to generate bidirectional interactive features;

[0065] S600, based on bidirectional interactive features, two-dimensional tensor representation of drug modality and component aggregation tensor, performs progressive fusion processing to generate fused feature tensor;

[0066] S700 outputs the fused feature tensor to the downstream task model.

[0067] The following is a detailed description of each step.

[0068] S100: Obtain the list of drug names and chemical component identifiers of traditional Chinese medicine prescriptions, perform embedded vector queries of drugs and components, and construct an aligned index structure.

[0069] The input data for this step includes a list of drug names for a traditional Chinese medicine formula and a list of chemical component identifiers corresponding to each drug in the formula. Specifically, the system receives a list of drug names for a formula containing n drugs. The system has a built-in mapping dictionary that maps drug names to drug identifiers, storing the correspondence between drug names and unique drug identifiers. The system queries the dictionary based on the list of drug names to obtain a sequence of drug identifiers of length n. The system also has a built-in drug-component mapping table that stores a list of chemical component identifiers corresponding to each drug identifier. The system queries the drug-component mapping table based on the sequence of drug identifiers to obtain a nested set of chemical component identifier lists, which contains n sublists, where the i-th sublist contains... Each chemical component identifier represents the chemical composition of the i-th drug.

[0070] After obtaining the list of chemical component identifiers, the system constructs an aligned index structure. The system calculates the start and end indices of the chemical component corresponding to each drug in the flattened sequence. The start index is then set. The value is 0. For the i-th drug (i ranges from 1 to n), its starting index is 0. It equals the sum of the chemical components of the first i-1 medicinal ingredients; its ending index equal In addition to the number of chemical components of the drug Subtract 1 again. The system stores the calculated index pairs as an aligned index set. .

[0071] S200. Based on drug embedding vectors and component embedding vectors, construct a two-dimensional tensor representation of drug modalities and a nested tensor representation of component modalities.

[0072] The input for this step is the drug embedding vector list, chemical component embedding vector list, and alignment index set output by S100. Specifically, the system stacks the drug embedding vector list in the original drug order to form a regular two-dimensional tensor, denoted as the drug modality tensor. , where n is the number of drugs and d is the embedding dimension.

[0073] For chemical components, the system constructs a nested tensor representation based on the alignment index set. The system iterates through i from 1 to n, and for the i-th drug, it uses its alignment index ( ), extract indices from slices in the flattened chemical composition embedded vector list. arrive (Including) 1 embedding vector, and this A d-dimensional vector is organized into a matrix. The system stores all n matrices as a set, denoted as the nested tensor representation of the component modes. This nested structure avoids padding sequences of different lengths to a uniform length.

[0074] S300. Perform adaptive aggregation processing on the nested tensor representation of the component modes to generate a component aggregation tensor;

[0075] The input for this step is the nested tensor representation of the component modes output by S200. Specifically, the system initializes an empty component aggregation tensor. The system iterates through i from 1 to n, examining the component matrix of each drug. Perform the following aggregation operations:

[0076] First, in the matrix A special, learnable aggregation marker [AGG] d-dimensional embedding vector is added to the front end to form an expanded matrix. .

[0077] Next, Input a component-level Transformer encoder. This encoder contains... The system consists of layers, each containing a multi-head self-attention mechanism and a feedforward neural network. The encoder encodes the extended sequence and outputs a hidden state matrix. In the sequence output by the encoder, the hidden state at the position corresponding to the aggregation marker [AGG] is extracted as the initial aggregation representation of the drug, denoted as... .

[0078] The system represents the initial aggregation. Perform gating residual enhancement. Calculate the gating factor:

[0079] ;

[0080] Computational Enhanced Aggregate Representation ,in, This is the initial aggregation representation. For the enhanced aggregation representation, , , , For learnable parameters, This represents element-wise multiplication. Finally, Assigned to the component aggregation tensor The i-th row. After traversing, It becomes a tensor of shape (n×d), where the i-th row represents the aggregation characteristics of all chemical components of the i-th drug.

[0081] S400: Based on the preset Chinese herbal medicine compatibility knowledge base and the drug list of the input prescription, calculate the compatibility prior bias matrix;

[0082] The input for this step is the drug identifier sequence obtained in S100 and the system's preset traditional Chinese medicine compatibility knowledge base. Specifically, the system constructs an n×n compatibility prior bias matrix Prior based on the drug identifier sequence, and initializes it to a zero matrix.

[0083] The system queries a pre-defined traditional Chinese medicine compatibility knowledge base. This knowledge base is a static table that stores the compatibility compatibility scores between any two herbs (using the herb identifier as the key). The score is based on the definitions in classic works such as "Shennong's Classic of Materia Medica" and "Compendium of Materia Medica". The quantification rules are as follows: mutual dependence score is 0.8, mutual assistance score is 0.6, mutual restraint score is 0.2, mutual antagonism score is 0.3, mutual aversion score is -0.4, opposite score is -0.8, and no clear relationship score is 0.0.

[0084] The system iterates through i from 1 to n and j from 1 to n. For each pair of drugs (the i-th and j-th drugs), it queries the knowledge base based on their drug identifiers to obtain a compatibility score. Then calculate the elements of the prior bias matrix: ,in This is a temperature coefficient used to adjust the intensity of prior information; the default value is 0.1.

[0085] S500, based on drug modality two-dimensional tensor representation, component aggregation tensor and compatibility prior bias matrix, performs bidirectional cross-attention calculation to generate bidirectional interactive features;

[0086] The input for this step is the drug modal tensor output by S200. The component aggregation tensor output by S300 And the matching prior bias matrix Prior output by S400. Specifically, the system first... and Perform linear projections on each to obtain their respective query (Q), key (K), and value (V) vectors: ;

[0087] After the calculation is completed, the complete matching prior bias matrix is ​​obtained. .in, , , , , , It is a learnable projection matrix. It is a dimension of attention mechanism.

[0088] Calculate the attention score from drug to ingredient: ,in This is a drug query vector. Here, RPE is the relative position encoding matrix, used to encode the relative order of drugs in the prescription, and Prior is the compatibility prior bias matrix; based on the attention score, the drug-to-component cross-attention output tensor is calculated. ,in Given a vector of component values; calculate the attention score from component to drug: ,in For component query vectors, The drug key vector is used; the component-to-drug cross-attention output tensor is calculated based on the attention score. ,in This is a vector of drug values. Ultimately, and As a bidirectional interactive feature output.

[0089] S600, based on bidirectional interaction features, two-dimensional tensor representation of drug modality and component aggregation tensor, performs progressive fusion processing to generate fused feature tensor;

[0090] The input for this step is the bidirectional interactive feature output by the S500. and Drug modal tensor output by S200 And the component aggregation tensor output by S300 Specifically, the system performs a three-stage fusion:

[0091] Phase 1: Element-level fusion. The system calculates two gate vectors: and ,in These are learnable parameters. Then, the element-level fusion result is calculated:

[0092] , where ⊙ represents element-wise multiplication.

[0093] Phase Two:

[0094] Semantic-level fusion: , and The concatenated feature tensor is obtained by concatenating the features. And compute the output tensor of semantic-level fusion. ,in , These are learnable parameters;

[0095] Residual Refinement: Computing the final fused feature tensor ,in This is a learnable fusion strength parameter, initialized to 0.1, used to control the fusion strength. LayerNorm is the layer normalization operation. This is the final deep fusion feature representation.

[0096] S700 outputs the fused feature tensor to the downstream task model.

[0097] This step will generate the final fused feature tensor from S600. As output. Specifically, the system will This is passed to a prediction head module relevant to a specific task, such as a fully connected layer classifier for predicting the efficacy of traditional Chinese medicine, or a regressor for predicting the potency of the medicine. The prediction head module is based on... The calculations are performed to generate the final prediction results. It integrates macroscopic drug information with microscopic chemical component information and preserves the hierarchical structure of drug-component, which can effectively improve the performance of downstream tasks.

[0098] Example 2

[0099] This embodiment illustrates a multimodal deep fusion system for traditional Chinese medicine features based on a dynamic tensor alignment mechanism. (Refer to the appendix.) Figure 5 The architecture consists of seven core functional modules arranged sequentially: a data processing module, a tensor construction module, an adaptive aggregation module, a priori computation module, a bidirectional cross-attention module, a progressive fusion module, and an output module. These modules are coupled through clearly defined data structures (tensors) and processing flows. The output of one module serves as the input of the next, forming an end-to-end computational pipeline from raw TCM data to deeply fused features. The following sections will describe the definition, function, detailed implementation methods of each module, and the connections and collaboration between them.

[0100] Data Processing Module. This module is the system's data input and alignment preprocessing front end. Its core responsibility is to receive structured descriptions of traditional Chinese medicine prescriptions, convert the text identifiers of drugs and chemical components into numerical embedding vectors, and accurately construct the index alignment relationship between drugs and their respective chemical components, providing a foundation for subsequent hierarchical tensor construction and fusion.

[0101] Detailed Implementation: During system initialization, three predefined data structures are loaded: a drug terminology, a chemical component terminology, and a drug-component mapping table. The drug terminology stores the names of common traditional Chinese medicines and their corresponding unique numerical identifiers. The chemical component terminology stores the names of known chemical components and their corresponding unique numerical identifiers. The drug-component mapping table uses the drug identifier as the key, and its values ​​are a list of chemical component identifiers contained in that drug.

[0102] When the module receives an input prescription, the input data includes a list of drug names and a list of corresponding chemical components.

[0103] First, drug identification is performed. The drug thesaurus is queried sequentially from the drug name list to obtain the drug identification sequence. Second, component identification and alignment index construction are performed. The drug-component mapping table is queried sequentially from the drug identification sequence to obtain the list of chemical component identifiers corresponding to each drug. Then, the position of each drug's chemical component in the flattened sequence is calculated based on the length of each list. The rule is: the starting index is the sum of the quantities of all previous drug components, and the ending index is the starting index plus the current drug component quantity minus one. The resulting alignment index set is as follows.

[0104] Finally, vector embedding is performed. Through two separate embedding layers, the drug identifier sequence is converted into a list of drug embedding vectors, and the flattened ingredient identifier list is converted into a list of ingredient embedding vectors. The dimension d of the embedding vectors is set to 256.

[0105] Inter-module connections: The output of this module includes a list of drug embedding vectors, a list of component embedding vectors, and a set of alignment indices. This data is encapsulated into a structured data object and passed directly to the tensor construction module.

[0106] Tensor Construction Module. This module is responsible for constructing a regular tensor representation of the vector list and alignment relationships provided by the upstream into the system's internal processing requirements. Its core is to generate a drug tensor representing the macroscopic structure of the prescription and a nested component tensor that preserves the attribution relationships of microscopic components.

[0107] Detailed implementation method:

[0108] The module receives a list of drug embedding vectors, a list of component embedding vectors, and a set of alignment indices from the data processing module.

[0109] First, a two-dimensional tensor for drug modalities is constructed. The list of drug embedding vectors is stacked in its original order to form a two-dimensional tensor. Secondly, construct the component modality nesting tensor. Slice from the flattened list of component embedding vectors based on the alignment index set. Inter-module connections: Output of this module. Directly provided to the bidirectional cross-attention module and the progressive fusion module. Nested component tensor set { This information is then provided to the adaptive aggregation module for further processing.

[0110] The adaptive aggregation module is responsible for intelligently aggregating the variable number of chemical component tensors corresponding to each drug, generating a fixed-dimensional aggregated representation for each drug that can summarize all its component information. It solves the normalization problem of variable-length sequences.

[0111] Detailed implementation method:

[0112] The module traverses the received nested component tensor set { For each component matrix in the set, perform the following operations:

[0113] exist An embedding vector of a learnable, 256-dimensional aggregate marker [AGG] is added to the front of the sequence to form an extended sequence matrix.

[0114] The extended sequence is fed into a component-level Transformer encoder. The encoder is configured with two layers, each containing four attention heads, and the feedforward network has a hidden layer dimension of 512.

[0115] Take the hidden state vector at the position corresponding to the aggregation mark in the encoder output sequence as the initial aggregation representation of the drug.

[0116] Subsequently, a gated residual mechanism is applied to enhance this representation. The gate factor is calculated. The enhanced aggregate representation is calculated. After processing all drugs, the resulting enhanced aggregate representations are stacked to form the component aggregate tensor. .

[0117] Inter-module connections: The component aggregation tensor output by this module It is passed to the bidirectional cross-attention module and the progressive fusion module as a post-aggregated feature representation of the component modalities.

[0118] Prior calculation module. This module is responsible for quantifying prior knowledge about drug compatibility in the field of Traditional Chinese Medicine into a numerical matrix, which is used to guide and constrain subsequent cross-modal attention calculations, thereby enhancing the model's domain rationality and generalization ability.

[0119] Detailed implementation method:

[0120] During system initialization, a pre-defined drug compatibility scoring table is loaded. This table, based on classic definitions such as those in the *Shennong Bencao Jing* (Shennong's Classic of Materia Medica), assigns a specific score to each pair of drugs. The module receives a list of drugs from the input prescription. The module then queries the compatibility scoring table based on the drug list. Based on the query results, a priori bias matrix for drug compatibility is constructed.

[0121] Inter-module connection: The matching prior bias matrix Prior generated by this module is output to the bidirectional cross-attention module as an additive bias term in the attention calculation of this module.

[0122] Bidirectional Cross-Attention Module. This module is the core of enabling deep interaction between drug and chemical component information. It utilizes attention mechanisms to allow the drug representation to query and absorb its component information, while simultaneously enabling the component aggregation representation to query and absorb contextual information about drug compatibility.

[0123] Detailed implementation method:

[0124] The module receives three inputs: drug tensor Component polymerization tensor Prior, the matching prior matrix.

[0125] right and Perform linear projections separately to obtain their respective query Q, key K, and value V vectors, maintaining a projection dimension of 256. Then, perform cross-attention calculations from drug to component. Cross-attention calculation of ingredients to drugs .

[0126] Inter-module connection: The two interaction tensors output by this module and It is passed to the progressive fusion module as a two-way interactive feature to be fused.

[0127] Progressive Fusion Module. This module is responsible for fusing bidirectional interactive features with original modal features in a stable and controllable manner at multiple levels, gradually refining the final multimodal fusion representation, and avoiding information loss or distortion.

[0128] Detailed implementation method:

[0129] The module receives five inputs: , Drug tensor Component polymerization tensor .

[0130] The first stage is element-level fusion. The second stage is semantic-level fusion. The third stage is residual refinement. The final fused feature tensor is then calculated.

[0131] Inter-module connections: The final fused feature tensor output by this module It is passed to the output module.

[0132] Output module. This module is the system's terminal, responsible for formatting the fused feature tensor and outputting it to the downstream task model, completing the final step from feature computation to application integration.

[0133] Detailed implementation method:

[0134] The module receives the final fused feature tensor from the progressive fusion module. .

[0135] Module pair Instead of performing any additional transformations, it is encapsulated into a standard data structure. This data structure contains the tensor data itself, as well as the associated metadata.

[0136] Overall system connection and operation process:

[0137] The entire system operates in a backend data stream driven manner. At the start of processing, the raw prescription data is input into the data processing module. The vectors and indices generated by this module are directly passed to the tensor construction module. The tensor construction module generates drug tensors in parallel. The nested component tensor sets flow directly to the bidirectional cross-attention module and the progressive fusion module, while the latter flows to the adaptive aggregation module. The adaptive aggregation module aggregates the nested sets into... Subsequently, it also flows to the bidirectional cross-attention module and the progressive fusion module. Simultaneously, the prior calculation module independently calculates the Prior matrix based on the input formula and flows it to the bidirectional cross-attention module. The bidirectional cross-attention module, upon receiving... , After Prior, computation is performed and the two interaction tensors are output to the progressive fusion module. The progressive fusion module, after gathering all the required features from multiple upstream modules, performs a three-level fusion computation to generate the final result. The output module delivers the data. The entire process is unidirectional and phased. Each module can perform calculations independently after receiving all the necessary inputs. The system collaborates through explicit data flow dependencies, ultimately achieving efficient and stable conversion from traditional Chinese medicine prescription text to deep multimodal fusion features.

[0138] It should be noted that, in this document, the terms "comprising," "including," and any other variations are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Specific examples have been used in this document to illustrate the principles and implementation methods of the present invention. These examples are merely for the purpose of helping to understand the method and core ideas of the present invention. The above descriptions are only preferred embodiments of the present invention. It should be pointed out that, due to the limitations of written expression and the objective existence of infinite specific structures, those skilled in the art can make several improvements, modifications, or variations without departing from the principles of the present invention, and can also combine the above technical features in an appropriate manner. These improvements, modifications, variations, or combinations, or the direct application of the concept and technical solution of the present invention to other situations without modification, should all be considered within the scope of protection of the present invention.

Claims

1. A multimodal traditional Chinese medicine feature fusion method based on dynamic tensor alignment, characterized in that, Includes the following steps: S100: Obtain the list of drug names and chemical component identifiers of traditional Chinese medicine prescriptions, perform embedded vector queries of drugs and components, and construct an aligned index structure; S200. Based on drug embedding vectors and component embedding vectors, construct a two-dimensional tensor representation of drug modalities and a nested tensor representation of component modalities. Step S200 specifically includes: stacking the drug embedding vectors in drug order to form a two-dimensional tensor; organizing the chemical component embedding vectors of each drug into matrices to form a nested tensor set; The input for this step is the drug embedding vector list, chemical component embedding vector list, and alignment index set output by S100; specifically, the system stacks the drug embedding vector list in the original drug order to form a regular two-dimensional tensor, denoted as the drug modality tensor. , where n is the number of drugs and d is the embedding dimension; For chemical components, the system constructs a nested tensor representation based on the alignment index set. The system iterates through i from 1 to n. For the i-th drug, it uses its alignment index ( ), extract indices from slices in the flattened chemical composition embedded vector list. arrive of 1 embedding vector, and this A d-dimensional vector is organized into a matrix. The system stores all n matrices as a set, denoted as the nested tensor representation of the component modes. This nested structure avoids padding sequences of different lengths to a uniform length. S300. Perform adaptive aggregation processing on the nested tensor representation of the component modes to generate a component aggregation tensor; The process enhanced via the gated residual mechanism in step S300 specifically includes: Obtain the aggregated token representation output by the component-level Transformer encoder. ; Calculate the gating factor ; Computational Enhanced Aggregate Representation ,in, This is a preliminary aggregation representation. For the enhanced aggregation representation, , , , For learnable parameters, This represents element-wise multiplication; the enhanced polymerization representations of all drugs are stacked into a component polymerization tensor. ; S400: Based on the preset Chinese herbal medicine compatibility knowledge base and the drug list of the input prescription, calculate the compatibility prior bias matrix; S500, based on drug modality two-dimensional tensor representation, component aggregation tensor and compatibility prior bias matrix, performs bidirectional cross-attention calculation to generate bidirectional interactive features; S600, based on bidirectional interaction features, two-dimensional tensor representation of drug modality and component aggregation tensor, performs progressive fusion processing to generate fused feature tensor; The process of sequentially performing element-level fusion, semantic-level fusion, and residual refinement in step S600 specifically includes: Element-level fusion: Calculate the output of element-level fusion. ,in , For learnable parameters, For the drug-to-component cross-attention output tensor, For the cross-attention output tensor from component to drug; Semantic-level fusion: , and The concatenated feature tensor is obtained by concatenating the features. And compute the output tensor of semantic-level fusion. ,in , These are learnable parameters; Residual Refinement: Computing the final fused feature tensor ,in LayerNorm is a learnable fusion strength parameter and a layer normalization operation. S700 outputs the fused feature tensor to the downstream task model.

2. The multimodal traditional Chinese medicine feature fusion method based on dynamic tensor alignment according to claim 1, characterized in that, The construction of the alignment index structure in step S100 specifically includes: Search the thesaurus based on the list of drug names to obtain the corresponding drug identifier sequence; The drug-component mapping table is queried based on the drug identifier sequence to obtain a list of chemical component identifiers corresponding to each drug. Based on the length of each chemical component identifier list, calculate the start and end indices of the chemical component corresponding to each drug in the flattened sequence to form an alignment index set.

3. The multimodal traditional Chinese medicine feature fusion method based on dynamic tensor alignment according to claim 1, characterized in that, The specific process of performing bidirectional cross-attention calculation in step S500 includes: For the drug tensor... and component polymerization tensor Perform linear projections to obtain the query vector, key vector, and value vector; calculate the attention score from drug to component. ,in This is a drug query vector. Let RPE be the component key vector, RPE be the relative position encoding matrix, and Prior be the matching prior bias matrix; calculate the drug-to-component cross-attention output tensor based on the attention score. ,in Given a vector of component values; calculate the attention score from component to drug: ,in For component query vectors, The drug key vector is used; the component-to-drug cross-attention output tensor is calculated based on the attention score. ,in This is a vector of drug values.

4. The multimodal traditional Chinese medicine feature fusion method based on dynamic tensor alignment according to claim 1, characterized in that, Step S400 specifically includes: querying a preset compatibility scoring table based on the drug list of the input prescription, obtaining the compatibility score between any two drugs, wherein the scoring table is based on the compatibility relationship definition in classic Chinese medicine works; and calculating the elements in the prior bias matrix. ,in For temperature coefficient, , It is a drug.

5. The multimodal traditional Chinese medicine feature fusion method based on dynamic tensor alignment according to claim 1, characterized in that, The process of generating aggregated markers through a component-level Transformer encoder in step S300 specifically includes: adding a special aggregated marker [AGG] to the front of the chemical component sequence of each drug to form an extended sequence; inputting the extended sequence into a component-level Transformer encoder, which includes a multi-head self-attention mechanism and a feedforward neural network; and taking the hidden state at the position corresponding to the aggregated marker [AGG] in the sequence output by the encoder as the initial aggregated representation of the drug.

6. The multimodal traditional Chinese medicine feature fusion method based on dynamic tensor alignment according to claim 4, characterized in that, In step S400, the compatibility scoring table is based on the definition of compatibility relationships in classic Chinese medicine texts, and its construction rule is: if two drugs have a mutually reinforcing relationship, then... =0.8; If the two drugs have an synergistic relationship, then =0.6; If the two drugs are incompatible, then =0.2; If the two drugs are antagonistic, then =0.3; If the two drugs are incompatible, then =-0.4; If the two drugs are inversely related, then =-0.8; if there is no explicit matching relationship, then =0.

0.

7. A multimodal traditional Chinese medicine feature fusion system based on dynamic tensor alignment, applied to the method of any one of claims 1-6, characterized in that, include: The data processing module is used to receive input data, obtain drug embedding vectors and component embedding vectors, and construct alignment indexes; The tensor construction module is used to construct two-dimensional tensor representations of drug modalities and nested tensor representations of component modalities; The adaptive aggregation module is used to adaptively aggregate the nested tensor representations of component modes to generate component aggregation tensors; The prior calculation module is used to calculate the compatibility prior bias matrix; the bidirectional cross-attention module is used to perform bidirectional cross-attention calculation between drugs and components. The progressive fusion module is used to progressively fuse bidirectional interactive features; The output module is used to output the final fused feature tensor; The output of the data processing module is connected to the tensor construction module; the output of the tensor construction module is connected to the adaptive aggregation module, the progressive fusion module, and the bidirectional cross-attention module; the output of the adaptive aggregation module is connected to the bidirectional cross-attention module and the progressive fusion module; the output of the prior calculation module is connected to the bidirectional cross-attention module; the output of the bidirectional cross-attention module is connected to the progressive fusion module; and the output of the progressive fusion module is connected to the output module.

Images