Referral decision method and system based on multi-modal fusion

Abstract

The present application relates to a referral decision-making method and system based on multi-modal fusion, adopting a technical architecture of multi-modal feature fusion + hierarchical prediction, comprising: acquiring multi-modal data under a referral scenario; respectively performing feature encoding on the multi-modal data to obtain feature vectors corresponding to the modes; and fusing the feature vectors to obtain multi-modal fusion features; based on the multi-modal fusion features, a plurality of second layer sub-dimension adaptation values are predicted; a plurality of first layer large category adaptation values are obtained by aggregating the plurality of second layer sub-dimension adaptation values; a comprehensive adaptation value is obtained by aggregating the first layer large category adaptation values; and a referral decision result is generated based on the comprehensive adaptation value.

Description

Technical Field

[0001] This invention relates to the field of medical artificial intelligence technology, and in particular to a referral decision-making method and system based on multimodal fusion. Background Technology

[0002] In the hierarchical medical system and the construction of medical consortia, patient referral is a crucial link in achieving the rational allocation of medical resources and ensuring the continuity of medical care. Currently, clinical referral decisions mainly rely on human experience, administrative orders, or the simple principle of proximity, resulting in problems such as crude matching, low efficiency, and uneven resource allocation. With the development of smart healthcare technology, existing technologies are gradually adopting machine learning, deep learning, and graph network models to achieve intelligent matching between patients and hospitals. By constructing a system of diagnostic and treatment indicators and assessing the compatibility between patient conditions and hospital resources, automated referral recommendations can be achieved, improving the efficiency and objectivity of referral decisions to a certain extent.

[0003] Chinese patent application number 202610049602.9 discloses an intelligent referral decision-making method for collaborative matching between patients and hospitals. This method constructs a patient-hospital-fitness triplet sample, builds fitness evaluation indicators from multiple dimensions including hospital qualifications, patient condition, geographical accessibility, and treatment continuity, and uses a heterogeneous network of medical consortia and a graph attention network to achieve feature aggregation and fitness calculation, ultimately outputting a comprehensive fitness score. While capable of making referral decisions, it still has significant technical shortcomings in its model architecture and practical applications. The data modality is singular and the feature representation is insufficient: it mainly relies on structured data and the topology of medical alliances, without integrating multimodal information such as medical text, diagnosis and treatment data, and referral policy codes, making it difficult to comprehensively depict the characteristics of the disease, resource status and referral constraints, thus affecting the final matching effect; The output was not hierarchically processed: (1) The hierarchical structure of the indicators was not explicitly output, resulting in insufficient observability and interpretability of the decision: Although a potential hierarchical structure was formed during the sample construction stage, the model only output a single comprehensive fitness score and did not explicitly output the quantitative results of each level and sub-dimension. Medical staff could not intuitively know the contribution of each indicator to the referral decision, resulting in opaque referral basis and difficulty in implementation and acceptance; (2) The score calculation process was completed in one step: The weighted aggregation process of each dimension indicator participated in the training and update together with the model parameters. The aggregation weight was learned by the model itself rather than fixed by the preset. The calculation rules of the comprehensive fitness were uncontrollable and unstable. The final output results were prone to deviation from the preset indicator logic and manual annotation standards, making it difficult to ensure the consistency and reliability of the referral decision. (3) The model directly fitted the comprehensive score end-to-end, resulting in complex training objectives and limited prediction accuracy: The model directly fitted the total score after fusing multiple levels of indicators. The task was complex, difficult to converge, and susceptible to noise interference. It could not be independently optimized for fine-grained sub-dimensions, further limiting the accuracy of fitness assessment.

[0004] Therefore, existing intelligent referral technologies cannot simultaneously meet the clinical needs of high precision, interpretability, fixed rules, and stable results, which is a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0005] To address at least one of the aforementioned technical problems, this invention provides a referral decision-making method based on multimodal fusion, comprising: S1: Acquire multimodal data in referral scenarios; S2: Encode the features of the multimodal data separately to obtain the feature vectors of the corresponding modes; and fuse the feature vectors to obtain the multimodal fused features; S3: Based on multimodal fusion features, predict several second-layer sub-dimension fit values; aggregate the several second-layer sub-dimension fit values ​​to obtain several first-layer major category fit values; aggregate the first-layer major category fit values ​​to obtain a comprehensive fit value; S4: Generate referral decision results based on the comprehensive fit value.

[0006] Furthermore, multimodal data, including semantic text modality, quantified numerical modality, medical consortium graph structure modality, and referral policy coding modality, are encoded to obtain text feature vectors, numerical feature vectors, graph attention vectors, and ontology path vectors. The steps for fusing these feature vectors to obtain multimodal fusion features include: The text feature vector and the numerical feature vector are concatenated and input into the gating weight calculation unit; the learnable weight matrix and the bias term are linearly transformed, and then the activation function is applied to generate adaptive gating weights. Based on adaptive gating weights, the text feature vector and numerical feature vector are weighted and fused to obtain the gated weighted text-numerical fusion feature. The graph attention vector is linearly transformed by a learnable projection matrix and mapped to a feature space consistent with the text-numerical fusion feature. Then, it is added to the text-numerical fusion feature to obtain the cross-type joint feature. By concatenating cross-type joint features with ontology path vectors and applying shared linear transformation and ReLU nonlinear activation, multimodal fusion features are obtained.

[0007] Furthermore, the semantic text modality is feature encoded, specifically: after word embedding, a bidirectional LSTM is used to obtain the hidden state sequence, and then a temporal attention pooling is used to obtain the semantic feature vector; The quantized numerical modes are feature-encoded, specifically by extracting numerical feature vectors through a combination of multilayer perceptron and segmented activation.

[0008] Furthermore, feature encoding is performed on the structural modalities of the medical consortium diagram, specifically as follows: First, by stacking multi-layer graph attention networks, multi-level attention weighted aggregation is performed on the institutional node features of heterogeneous subgraphs in the medical consortium graph structure; Then, a graph-level mean-maximum concatenation operation is performed on the aggregated node features. The global mean and global maximum values ​​of the node features are calculated and concatenated to merge the global statistical features and extreme value features of the subgraph. Finally, the concatenated features are mapped into a fixed-dimensional graph attention vector through a linear projection layer.

[0009] Furthermore, feature encoding is performed on the referral policy coding modality, specifically by embedding the policy code into a pre-set fixed vocabulary to generate an ontology path vector.

[0010] Further, in step S3, the first-level category fit value and overall fit value are obtained by aggregating using preset formulas (1)-(2): (1); (2); in, This represents the number of sub-dimensions in the second layer. The number of items in the first-level category. For the first The first of the major categories Weights of each sub-dimension For the first The second-level sub-dimension fit value For the first The set of sub-dimensions contained in each major category. for The first-level category fit value For the first Weights of each major category, This is the overall fit value; among which, , Pre-set.

[0011] Further, in step S4, formula (3) is used to determine the referral priority level based on the comprehensive fit value: (3); in, For the first Thresholds for each level, For the number of levels, Priority for referral; Alternatively, construct an integer linear programming model with a set of candidate receiving institutions. Institutions in China To define the decision object, introduce 0-1 decision variables. , Indicates that the organization Selected as the sole primary recommender recipient; The optimization objective is to maximize the weighted sum of the target score and the corresponding decision variables, based on the target score determined by the second-level sub-dimensional fitness value, the comprehensive fitness value, and the institutional characteristics. Set constraints: the sum of decision variables for all institutions is 1, the decision variable for each institution is no greater than its corresponding feasibility indicator variable, and all decision variables take the value of 0 or 1; the optimal solution of the integer linear programming model is used as the final primary recommending institution.

[0012] Furthermore, a referral decision model is constructed and trained, including: an input layer, a feature extraction layer, a hierarchical prediction layer, and an output layer, which are used to execute steps S1-S4 respectively; the training process of the referral decision model adopts a multi-level loss function, including: Based on the predicted and labeled values ​​of the fitness scores of the second-level sub-dimensions, the mean squared error loss is constructed as the second-level sub-dimension loss; based on the predicted and labeled values ​​of the fitness scores of the first-level major categories, the L2 loss is constructed as the first-level major category loss; based on the predicted and labeled values ​​of the comprehensive fitness scores, the mean squared error loss is constructed as the comprehensive fitness score loss. The total loss of the referral decision model is obtained by weighting the second-level sub-dimension loss, the first-level category loss, and the comprehensive fit value loss.

[0013] Furthermore, the multi-level loss function also includes: constructing a marginal loss as a level consistency loss based on the predicted and labeled values ​​of the comprehensive fitness value; constructing a direction alignment loss between text features and numerical features by constraining the changing trends of the disease course text features and the numerical features related to the urgency of the emergency; constructing an intra-group order constraint loss by applying soft order constraints to the sub-dimensions within a certain major category; and constructing a feature supervision loss by constructing a feature supervision loss between the dual-path feature output and the label. The total loss of the referral decision model also includes additional weighted summation: rank consistency loss, direction alignment loss, intra-group order constraint loss, and feature supervision loss.

[0014] On the other hand, the present invention also provides a terminal device, including a memory and a processor; the memory stores program code that can be executed by the processor; the program code is used to execute any of the above-mentioned referral decision methods.

[0015] This invention relates to a referral decision-making method and system based on multimodal fusion. It employs a technical architecture of multimodal feature fusion and hierarchical prediction, and compared to existing technologies, it offers at least the following significant technical advantages: comprehensive multimodal data representation, significantly improving the accuracy of fit prediction; explicit hierarchical prediction output, enabling observable and interpretable referral decisions; fixed pre-set hierarchical aggregation weights that do not participate in training, ensuring rule stability and controllable results; model training only at fine-grained sub-dimensions, resulting in faster convergence, higher accuracy, and greater stability; and hierarchical output that better aligns with clinical approval, quality control, and regulatory needs. Attached Figure Description

[0016] Figure 1 This is a flowchart of one embodiment of the referral decision-making method of the present invention. Detailed Implementation

[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0018] It should be noted that if the embodiments of the present invention involve directional indications, such as up, down, left, right, front, back, etc., these directional indications are only used to explain the relative positional relationships and movement of the components in a specific posture. If the specific posture changes, the directional indications will also change accordingly. Furthermore, if the embodiments of the present invention involve descriptions such as "first," "second," "S1," "S2," "step one," "step two," etc., these descriptions are for descriptive purposes only and should not be construed as indicating or implying their relative importance, or implicitly indicating the number of technical features indicated or the execution order of the method. Those skilled in the art will understand that anything that does not violate the inventive concept should be included within the scope of protection of the present invention.

[0019] like Figure 1 As shown, this invention provides a referral decision-making method based on multimodal fusion, comprising: S1: Acquire multimodal data in referral scenarios; Specifically, when issuing a referral request, the following data can be provided: textual data such as the patient's past medical records, chief complaint, and current medical history; quantitative numerical data such as age, weight, blood pressure, and laboratory indicators; graph structure data such as the topology of medical consortium institutions and referral relationship paths; ontological data of referral policies such as the hierarchical diagnosis and treatment catalog, referral direction, path type, and policy code; and data on the operational status of institutions such as the distribution of beds and equipment, distance, waiting time, and conditions for medical record sharing.

[0020] In a preferred embodiment, the multimodal data may optionally include any two or more of the following: semantic text modality, quantitative numerical modality, medical consortium diagram structure modality, referral policy coding modality, and perception time-series modality. It is worth noting that these five modalities are merely illustrative examples of multimodal data and are not limited to them; some modalities may be included, or other modalities may also be included. Specifically, an input layer may be optionally configured to receive the following inputs: (1) Semantic text modality, which can be the core business input modality, and may include: referral reasons, disease progress summary, medical orders, follow-up points, speech-to-text and other natural language texts related to referral business; the input rule is the actual input, and the upper bound of the sequence length. The vocabulary size is limited; when there is no text data, empty strings are used as placeholders and encoded as zero vectors. (2) Quantitative numerical modalities can be used as input modalities for core business operations. Options include: vital signs and test results (age, blood pressure / blood sugar, etc.), referral distance and cost, institutional load, route type, length of hospital stay, and other discrete numerical values. The input rules are the actual inputs and can be structured field vectors. When a value is missing, zeros are added as placeholders and normalization is performed. (3) Medical consortium graph structure modality, which can be a scenario-specific graph modality. Optional features include heterogeneous subgraphs of the medical consortium that match the treatment needs of referred patients. The subgraphs use medical institutions as nodes and referral channels or collaborative relationships as edges; they represent the hierarchical relationships between hospitals, departments, and doctors, the referral flow network, collaborative weights, and the regional medical resource topology. The input rules are actual input / placeholders, and the number of nodes in the subgraph is set to a fixed upper bound. The edge and node features have finite dimensions; when there is no graph data or the edge set is empty, zero node features and empty edge sets are used as placeholders. (4) Referral policy coding modality is an independent rule modality used to identify the discretized policy codes for the compliance and type of referral behavior. It directly reflects the policy rules of "whether or not to refer, where to refer, and what path to take," and is the key basis for judging whether the referral meets the requirements of medical insurance and hierarchical diagnosis and treatment. Optional referral directions include (coded according to the flow of medical institutions at different levels as stipulated by the policy; for example: 01: Upward referral (upward transfer): primary care → secondary care, secondary care → tertiary care, primary care → tertiary care (cross-level); 02: Downward referral (downward transfer): tertiary care → secondary care, secondary care → primary care, tertiary care → primary care (rehabilitation / chronic disease); 03: Horizontal referral (horizontal referral): between hospitals of the same level (same level, same region, within the medical consortium); 04: Cross-regional referral: cross-medical consortium within the city, cross-city within the province, cross-provincial inter-regional referral), and path types (coded according to the referral process and compliant path stipulated by the policy, directly related to the medical insurance reimbursement ratio; for example: 01: step-by-step referral). 01: Primary care referral (compliant): Primary care → Secondary care → Tertiary care (compliant with the "primary care first visit" principle of the hierarchical medical system); 02: Cross-level referral (special approval): Primary care directly → Tertiary care (limited to acute and critical illnesses / difficult cases, requires approval); 03: Referral within a medical consortium: Internal transfer within a closely integrated medical consortium / medical group (green channel); 04: Non-standard referral (out-of-pocket / reimbursement reduction): Directly bypassing the level without following the policy path / restricted by medical insurance in other regions), Hierarchical medical system catalog number (each number corresponds to "fixed disease + fixed level of receiving institution + fixed referral criteria"; for example: catalog number 001: Hypertension grade 1-2 (ICD-10: I10) → Primary care first visit; catalog number 057: Acute myocardial infarction (ICD-10: I21) → Tertiary hospital treatment, can be transferred upwards; catalog number 123: Cerebral infarction recovery period → Secondary / primary rehabilitation, should be transferred downwards) etc. discrete categories; input rules are actual input, and default values ​​are added as placeholders when there is no code; (5) Medical perception and temporal modality, which is reserved for extended modality, including: medical imaging (CT / MRI / DR / ultrasound), medical voice (raw audio), long-term monitoring (ECG / blood oxygen / blood pressure continuous temporal sequence), etc.; the input rule is to use placeholder access, and when there is no actual data, the output is a zero vector with the same shape as the feature vector to maintain the uniformity of the network interface; S2: Encode the features of the multimodal data separately to obtain the feature vectors of the corresponding modes; then fuse the feature vectors to obtain the multimodal fused features; specifically, the feature extraction layer can be optionally configured, including a first encoding unit, a second encoding unit, a third encoding unit, a fourth encoding unit, a fifth encoding unit, and a fusion unit: (1) The first encoding unit encodes the features of the semantic text modality. The narrative tower-oriented sequence evidence for pathological text can be selected: word embedding is followed by bidirectional LSTM to obtain the hidden state sequence. Then, the semantic feature vector is obtained through time-dimensional attention pooling. This semantic feature vector can carry the second layer of clinical and process-related semantic information; during the training period, random dropout can be applied to the LSTM and pooling paths to reduce overfitting, with a dropout probability of 0.1 in the example.

[0021] (2) The second encoding unit performs feature encoding on the quantized numerical modes. It can be selected to use a statistical tower for structured numerical data, and extract numerical feature vectors through a combination of multilayer perceptron (MLP) and piecewise activation. This numerical feature vector can characterize resources and spatiotemporal proxy quantities; random dropouts can be applied to each hidden layer of the multilayer perceptron during the training period, with a dropout probability of 0.1 in the embodiment.

[0022] (3) The third encoding unit encodes the features of the medical consortium graph structure modality. Optionally, it can be done by: firstly, stacking multi-layer graph attention networks (GAT) to perform multi-level attention weighted aggregation on the institutional node features of heterogeneous subgraphs in the medical consortium graph structure (fully exploring the referral and collaboration association features between nodes); then, performing graph-level mean-maximum concatenation operation on the aggregated node features, calculating the global mean and global maximum of the node features respectively and concatenating the features to fuse the global statistical features and extreme value features of the subgraphs; finally, mapping the concatenated features to a fixed-dimensional graph attention vector through a linear projection layer. Take the subgraph when it is empty. .

[0023] (4) The fourth coding unit performs feature coding on the referral policy coding modality. Optionally, the policy coding is embedded through a preset fixed vocabulary to generate an ontology path vector. The embedding process is a fixed setting and does not participate in model training. The model distinguishes between different referral policy contexts.

[0024] (5) The fifth encoding unit encodes the features of medical perception and temporal modality. It can optionally reserve a dedicated interface to adapt to medical image CV model, speech MFCC+CNN model, long temporal LSTM / TCN model, etc.

[0025] In this embodiment, a preferred embodiment of feature encoding is given for the data characteristics of each modality, but it is not limited thereto; those skilled in the art can select an encoding method suitable for the data characteristics according to the specific content of the multimodal data to obtain the feature vectors of each modality; and then perform fusion processing on the feature vectors of each modality.

[0026] In a preferred embodiment, the fusion unit: the step of fusing the feature vectors to obtain multimodal fused features may optionally include: Text feature vectors With numerical eigenvectors Feature concatenation is performed, and the input is given to the gating weight calculation unit; a learnable weight matrix is ​​then used. With bias term Perform a linear transformation, then pass through the first activation function. The option to use sigmoid can be used to generate adaptive gating weights. , is represented as: ; Based on adaptive gating weights, the text feature vectors are weighted and fused. With numerical eigenvectors The gated weighted text-numerical fusion features are obtained and represented as follows: ; Through a learnable projection matrix A linear transformation is performed on the graph attention vector to map it to a feature space consistent with the text-numerical fusion feature. Then, it is added to the text-numerical fusion feature to obtain the cross-type joint feature. , is represented as: ; Cross-type joint features With ontology path vector By concatenating the features and applying a shared linear transformation and ReLU nonlinear activation, further multimodal fusion features are obtained. , is represented as: .

[0027] Step S2, as the second inventive point of this application, is key in two parts: 1. the order of four-modal fusion; 2. the differentiated fusion methods between each modality. Specifically: 1. Four-modal fusion sequence: The fusion follows a book order of text + numerical gating fusion → graph feature space alignment → ontology path vector fusion. The core logic of this design adheres to the approach of "aggregation of similar types → deep cross-type interaction → supplementation of specific information." Text and quantitative numerical modalities are fused first because both belong to the basic attribute data of the institution, describing the basic information and business indicators of the medical consortium. They have high data dimensionality and semantic relevance, belonging to the same level of basic features. Fusion of these first forms a unified basic feature expression, reducing the computational complexity of subsequent multimodal overlay. Next, the medical consortium graph structure modal is incorporated, representing the topological relationships and interconnected networks between institutions. This is a relational high-order feature, belonging to a different information dimension than the basic attribute features. After the basic features are formed, graph features are then integrated to achieve a deep combination of "attribute information + relational information." Finally, the referral policy code (ontology path) modality is an upper-level business rule information that is attached to the institution's attributes and associations. As a supplementary feature, it is fused later to ensure that the basic features and network relationship subject information are not interfered with by the rule information, and to allow the final fused feature to carry the three complete types of information: basic attributes, network associations, and business rules.

[0028] 2. Differentiated fusion methods between different modalities: A: Text + numerical values, using gated weighted fusion. By dynamically allocating the contribution ratio of the two types of features using learnable weights, redundancy is adaptively suppressed and effective information is enhanced. This addresses the issue of the dynamic changes in the importance of textual semantics and quantitative numerical features with different business scenarios, improving the flexibility of fusion.

[0029] B: Graph attention vector + text-numerical fusion features, using linear projection + element-wise addition. First, feature space alignment is completed through the projection matrix to eliminate the differences in dimensionality and distribution between graph modalities and text-numerical fusion features; then, element-wise addition is used to achieve deep information interaction, allowing graph topological association information to be deeply coupled with text-numerical information.

[0030] C: Joint features + ontology path vectors, using concatenation + shared linear transformation + ReLU activation. The concatenation operation fully preserves the independent feature information of the referral policy ontology path, avoiding feature overwriting; subsequent unified linear transformation and nonlinear activation complete global feature encoding, further mining multimodal implicit associations and strengthening the overall representation capability.

[0031] In summary, the aforementioned four-modal fusion order and differentiated fusion methods are the result of the applicant's creative efforts. This solution targets four heterogeneous modalities of data within a medical consortium scenario: semantic text, quantitative values, graph structures, and referral policy codes. It employs a differentiated fusion strategy to integrate multimodal features, fully leveraging the unique information of each modality and preserving the original characteristics of the data while achieving effective complementarity between modalities. Through a combination of adaptive gating, spatial alignment, element-wise addition, and feature concatenation, the solution improves the accuracy and robustness of feature representation, reduces feature interference between heterogeneous modalities, and ultimately outputs multimodal fusion features with stronger discriminative power, adaptable to the reasoning and analysis tasks of medical consortium operations.

[0032] S3: Based on multimodal fusion features, predict several second-layer sub-dimension fit values; aggregate the several second-layer sub-dimension fit values ​​to obtain several first-layer major category fit values; aggregate the first-layer major category fit values ​​to obtain a comprehensive fit value; Preferably, based on multimodal fusion features or further integration of multimodal fusion features The second-layer sub-dimension fitness value is first predicted using a pre-defined hierarchical prediction layer. For example, based on the shared layer output, the second-layer linear mapping and sigmoid activation function are used to obtain the second-layer sub-dimension fitness value, expressed as: ; More preferably, the preset hierarchical preset model can be expressed as formulas (1) and (2): (1); (2); in, This represents the number of sub-dimensions in the second layer. The number of items in the first-level category. For the first The first of the major categories Weights for each sub-dimension (weights can be equal). For the first The second-level sub-dimension fit value (predicted). For the first The set of sub-dimensions contained in each major category. for The first-level category fit value For the first Weights of each major category, This is the overall fit value.

[0033] It is worth noting that the preferred embodiments of the calculation formulas for the first-level category fit degree value and the comprehensive fit score in formulas (1) and (2) can also be expressed as: , Other forms, such as the first-level category fit value and the comprehensive fit score, are not crucial in step S3. The key lies not in the specific determination of the first-level category fit value and the comprehensive fit score, but in the fact that it does not directly output the comprehensive fit value based on multimodal fusion features. Instead, it achieves fine-grained modeling and hierarchical reasoning of referral fit through a hierarchical prediction structure: second-level sub-dimension → first-level category → comprehensive fit value. First, the second level predicts each independent sub-dimension indicator separately, accurately depicting the matching degree of single dimensions such as patient condition, policy compliance, institutional capabilities, and path rationality, improving the model's sensitivity and discriminative ability to local features. Then, the sub-dimension scores are aggregated to the first-level category indicators, achieving the summarization and noise reduction of similar features, highlighting key influencing factors. Finally, the comprehensive fit value is output, providing a unified decision-making basis while retaining fine-grained interpretability. Its core functions include at least: (1) improving interpretability: each layer score corresponds to understandable business indicators, making it easier to trace the basis of referral decisions; (2) enhancing model stability: hierarchical optimization reduces the difficulty of end-to-end direct prediction and reduces mutual interference between different dimensions; (3) strengthening generalization ability: independent learning of sub-dimensions can better adapt to different disease types, different policies, and different medical alliance structures; (4) facilitating the embedding of policies and rules: prior knowledge such as hierarchical diagnosis and treatment catalogs and referral path constraints can be flexibly introduced at each layer; (5) improving the accuracy of final decision-making: through hierarchical aggregation from fine to coarse, the overall fit is more in line with the actual clinical referral logic.

[0034] In a preferred embodiment, as shown in Table 1, examples of the first-level categories and the second-level sub-dimensions are given. The first level consists of four macro categories, each of which includes three sub-dimensions. The weights of the categories and dimensions can be preset.

[0035] Table 1, Example Table of First-Level Category and Second Sub-Dimension:

[0036] Specifically, during the annotation of the training dataset, the annotation values ​​obtained are the fitness values ​​of the second-level sub-dimension. ;Depend on Through the continuation of fixed rule chains with the same structure as equations (1)-(2), aggregation is obtained. These are the labeled values ​​of the first-level category fit and the comprehensive fit, respectively. The results are then obtained according to formula (3). The referral outcome is determined; in the prediction phase, the same predictions are obtained through the network. Obtained through the same fixed rule chain This makes the prediction results more compatible with the previous annotations, achieving the effect of hierarchical prediction.

[0037] S4: Generate referral decision results based on the comprehensive fit score. Specifically, the final referral decision can be determined by ranking the comprehensive fit scores.

[0038] In a preferred embodiment, the referral priority level can be determined based on the comprehensive fit value. , is represented as: (3); in, For the first Thresholds for each level, Number of levels; Example: , , indicating the priority level of discretization, by With filing threshold Segmented, optional: .

[0039] More preferably, step S4 may also include integer linear programming for ranking candidate institutions: [The text abruptly ends here, likely due to an incomplete sentence or missing information.] And obtained through equations (1)-(3) Subsequently, if it is necessary to determine a unique primary recommender from the set of candidate receiving institutions, an operations research layer model can be constructed after decoupling from the deep network, specifically including: S41: Construct an integer linear programming model with a set of candidate receiving agencies. Institutions in China To define the decision object, introduce 0-1 decision variables. , Indicates that the organization Selected as the sole primary recommender recipient; S42: Based on the fit value of the second-layer sub-dimension Overall fit value and institutional characteristics The determined target score To optimize the objective, the goal is to maximize the weighted sum of the objective score and the corresponding decision variables; S43: Set constraints: The sum of the decision variables of all institutions is 1 to ensure that the output is a unique primary recommender institution; the decision variable of each institution is no greater than its corresponding feasibility indicator variable. To meet the hard constraints of policy, referral channels, and institutional capacity; and all decision variables take values ​​of 0 or 1; the optimal solution of the integer linear programming model. As the final primary recommending organization.

[0040] Specifically: The integer linear programming model can be represented as follows: Sets and Subscripts: Let The set of candidate receiving institutions (in this example, it can be screened by medical consortium subgraph nodes, policy whitelist, and geographical accessibility, etc.)

[0041] Decision variables: , ; Indicates that the organization As the sole primary recommender recipient.

[0042] Objective function and constraints: The objective function is... This serves as a unique primary recommendation constraint. As a constraint on candidate feasibility; The constraint is a 0-1 integer field.

[0043] Optimal solution: Optimal solution No need to modify or It can be used in conjunction with lexicographical order or weighted and extended values.

[0044] In a preferred embodiment, the method further includes: constructing and training a referral decision model, including: an input layer, a feature extraction layer, a hierarchical prediction layer, and an output layer, for respectively executing steps S1-S4; the training process of the referral decision model employs a multi-level loss function, including: Second-level sub-dimension loss: Based on the predicted and labeled values ​​of the second-level sub-dimension fit values, the mean squared error loss is constructed as the second-level sub-dimension loss. , is represented as: Where M is the number of samples; First-level category loss: Based on the predicted and labeled values ​​of the first-level category fitness values, construct the L2 loss as the first-level category loss. , is represented as: ; Overall fitness value loss: Based on the predicted and labeled overall fitness values, a mean squared error loss is constructed as the overall fitness value loss. , represented as ; By weighted summation , , The total loss was obtained. A more preferred multi-level loss function also includes: Level Consistency Loss: Based on the predicted and labeled values ​​of the overall fitness score, a marginal loss is constructed as the level consistency loss. , is represented as: The example uses a marginal constant. .

[0045] Orientation Alignment Loss: By constraining the variation trends of the disease course text features and the numerical features related to the urgency of the emergency to be consistent, an orientation alignment loss is constructed between the text features and the numerical features. Specifically: the urgency reflected in the alignment loss constraint pathology narrative aligns with the direction of the testable numerical proxy: assuming that after narrative tower pooling and linear projection, we get... ,from Take slices of numerical columns related to urgency and other factors. , If it is a small constant, then ; when All zero or Norm below At that time, the current sample can be... Set the term to zero or skip it to avoid meaningless gradients.

[0046] Intra-group order constraint loss: This involves applying soft order constraints to sub-dimensions within a given major category to construct the intra-group order constraint loss. For example, for the first major category In the set, satisfying Soft order constraints are applied to the samples, and the example uses an interval constant. The loss constrained by the order relation within the group is constructed as follows: ; Supervision loss for dual-path feature output and label construction Example: Dual-path Add when enabled .

[0047] Therefore, the total loss can be expressed as: ; in, The weights for each loss are determined by the fact that if a loss is not adopted, the corresponding item is set to zero when it is not enabled.

[0048] In this embodiment, the referral decision model employs a multi-level loss mechanism to minimize the fit error of the second-level sub-dimension during the training phase, and constrains the... Obtained from equations (1) and (2) respectively with Consistent, at the same time and By applying order-related marginal terms and including directional alignment terms between narrative embeddings and related numerical slices such as urgency, and by setting intra-group order relation constraint loss and feature supervision loss, the total loss is determined by weighting, which can further improve the accuracy of referral decisions.

[0049] The above describes the multimodal fusion-based referral decision-making method of this invention. After proposing a referral request, multimodal data is acquired, such as text sequences, numerical vectors, institutional subgraphs, and policy fields of a single sample. Multimodal fusion features are obtained through various branch encodings such as narrative towers and statistical towers, and then output. During training, only the network parameters are updated according to the loss function of each dimension. During inference, training period regularization such as random discarding is disabled, and fixed rules are applied, such as the formula chain (1)-(3), to obtain the results. ; When making specific decision outputs, it can be decoupled from the deep learning framework. If the main recommendation of the institution is enabled, the filing parameters and Calculation The algorithm solves an integer linear programming model to obtain the optimal solution for the primary recommendation result. Optionally, during batch inference, samples within the same batch share the same set of network weights and configuration parameters; if the number of graph nodes differs among samples within the same batch, nodes and edges can be padded and masked to ensure the batch processing shape of the graph neural network is consistent with the training phase. More optionally, the deep network uses half-precision floating-point in the forward pass to reduce memory usage and latency, and the connection to fixed rules... and threshold Comparative calculations should be performed at higher precision to reduce level boundary jitter. Alternatively, during inference, regularization techniques such as random dropout, which are only enabled during training, can be disabled. If the network contains runtime-dependent layers such as batch normalization, statistics accumulated during training can be used during inference. The deployment environment and training environment can differ in precision and device type. Accuracy metrics should be written based on reproducibility with fixed data and seeds, while the inference wall-clock can be a single measurement.

[0050] It is worth noting that in the improvement process of the referral decision-making method based on multimodal fusion in this invention, many comparative and ablation experiments were also conducted, including but not limited to: closing graph branches, changing gated fusion to multilayer perceptron after vector concatenation, disabling the loss terms related to the first layer and the comprehensive fit value and only training the second layer mean square error, setting the ontology vector to zero, and disabling the text-value alignment term, etc., to observe the contribution of each module and loss term to the indicators and stability. After the experiments, it was found that the above-mentioned improvements are not conventional technical choices and will greatly reduce the prediction accuracy, making them unsuitable for practical deployment. Compared with simple process informatization or one-way referral scheduling: This application jointly encodes the medical history text, structured values, and institutional collaboration graph. Under the unified two-layer indicators and the fixed rule chain of continuity, it can output verifiable sub-items and comprehensive conclusions, which is convenient for management benchmarking and record keeping. Compared with schemes that only rely on rules or shallow concatenation, this application uses bidirectional LSTM, graph attention and gated fusion, and ontology path vectors to establish a correspondence between the evidence type and the second-layer sub-dimensional fit value in the referral scenario, thereby improving the specificity of the representation. Compared with in-sample dynamic entropy weight aggregation, in this application Pre-deployment calibration and sample constraining, along with decoupling from network parameters, facilitate consistency with manual annotation and cross-institutional reproducibility. Even though existing solutions involve multi-path encoding and fusion in their implementation, this application, in its output structure, uses a unique, directly output second-layer sub-dimensional vector and a hierarchy following a fixed rule chain as the main path. In its training structure, it uses multi-level joint constraints and text-value alignment consistent with the scale, fundamentally different from generic multimodal fusion schemes that only perform end-to-end single-objective regression or single-label classification. In this invention, multiple objectives in clinical and management aspects are addressed through the second-layer sub-dimensional adaptation value. The overall fitness value is obtained by explicit expansion and then by fixed scalar weighted summation of equations (1) and (2). This belongs to the scalarization path in multi-objective decision-making; specifically, under the constraint of a continuous fixed rule chain, the network only learns the fitness values ​​of the second-layer sub-dimensions. The fitness values ​​of the first-layer major categories, the overall fitness values, and the levels are all obtained from a pre-defined, consistent fixed aggregation, which helps reduce the risk of rule drift. If it is necessary to emphasize a certain type of objective, it can be adjusted... and To perform sensitivity analysis without changing the network structure and parameters, only output... The overall structure of the fixed rule chain. None of these are learned parameters for deep networks, and their paths are completely different from the dynamic entropy weight re-aggregation within samples to obtain the total fitness score. If management needs to emphasize a certain type of objective, it can do so by updating the registration. This version performs sensitivity analysis without retraining the entire network, distinguishing it from the path of "automatically recalculating entropy weights per sample in the forward pass".

[0051] More preferably, to improve numerical stability and gradient isolation: before entering equation (1), the following can be performed: exist Upper clipping is used to suppress floating-point out-of-bounds errors; if the output of equations (1) and (2) slightly exceeds the limit due to rounding... Further trimming is possible. Training is underway; registration required. It is not registered with the optimizer as a learnable tensor. right The gradient can be backpropagated using automatic differentiation, or the loss can be expressed analytically. The gradient is obtained through the chain rule of formulas (1)-(3). In the embodiment where the main recommendation of the institution is enabled, the main recommendation is determined by an integer programming or equivalent interface decoupled from the network under the filing parameters. It can be audited separately from the sub-item prediction and the comprehensive level link, which makes it easy to explain why the institution is recommended at the management end, instead of relying entirely on the black box output for the selection of the institution.

[0052] In summary, the referral decision-making method based on multimodal fusion of this invention adopts a technical architecture of multimodal feature fusion + hierarchical prediction (preferred scheme uses preset fixed weight aggregation), and compared with the prior art, it has at least the following outstanding technical effects: Comprehensive multimodal data representation significantly improves the accuracy of fit prediction: This application integrates multimodal data such as text, numerical values, heterogeneous graphs, and policy codes, which can realize the full-dimensional feature representation of disease, resources, spatiotemporal, and policy, thereby improving the accuracy of matching from the data level.

[0053] The hierarchical prediction of explicit output enables observable and interpretable referral decisions: the model first predicts the fitness values ​​of the second-level sub-dimensions, then aggregates them to obtain the fitness values ​​of the first-level major categories, and finally calculates the comprehensive fitness value. The hierarchy is clear and the results are traceable, completely solving the "black box decision-making" problem of existing technologies.

[0054] Hierarchical aggregation weights are preset to be fixed and do not participate in training, ensuring rule stability and controllable results: The first-level aggregation and the final comprehensive calculation use fixed weights and fixed links, and do not participate in model training and backpropagation, so that the output results strictly follow the preset referral rules, are highly consistent with manual annotation, and greatly improve reliability and traceability.

[0055] The model is trained only on fine-grained sub-dimensions, resulting in faster convergence, higher accuracy, and greater stability: by breaking down complex tasks into simple sub-tasks, the model training objectives are clear, interference is minimal, and optimization is easy. Compared to end-to-end fitting of the total score, the generalization ability and prediction stability are significantly enhanced.

[0056] The tiered output is more in line with the needs of clinical approval, quality control and supervision: the tiered scores can be directly used for referral review, pathway management, medical insurance cost control and medical consortium supervision, and have stronger practicality.

[0057] In general, when primary care or specialized institutions initiate referral requests within a medical consortium network, the system receives patient text, numerical data, institution sub-maps, policy codes, etc., and outputs predictions aligned with the second-level sub-dimension fit values. These predictions are then processed through a fixed rule chain to obtain the first-level category fit value, overall fit value, and referral priority level, forming a verifiable referral plan representation. For preferred cases, an integer linear programming approach is used to obtain the primary recommendation from the set of candidate receiving institutions. The application scenarios and targets can be: managing multi-objective trade-offs in cross-institutional referrals within a medical consortium; the user-visible conclusion is a referral plan representation, including at least the second-level sub-dimension fit value prediction, the first-level category fit value, the overall fit value, and the referral priority level—a subset required for business operations—for display, benchmarking, and drill-down; in the embodiment, referrals can be executed based on physician confirmation. This invention outputs decision support information rather than solely replacing clinical decisions.

[0058] On the other hand, the present invention also provides a computer storage medium storing executable program code; the executable program code is used to execute any of the above-mentioned referral decision methods.

[0059] On the other hand, the present invention also provides a terminal device, including a memory and a processor; the memory stores program code that can be executed by the processor; the program code is used to execute any of the above-mentioned referral decision methods.

[0060] For example, the program code can be divided into one or more modules / units, which are stored in the memory and executed by the processor to complete the present invention. The one or more modules / units can be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the program code in the terminal device.

[0061] The terminal device can be a desktop computer, laptop, handheld computer, or cloud server, etc. The terminal device may include, but is not limited to, a processor and memory. Those skilled in the art will understand that the terminal device may also include input / output devices, network access devices, buses, etc.

[0062] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0063] The memory can be an internal storage unit of the terminal device, such as a hard drive or RAM. The memory can also be an external storage device of the terminal device, such as a plug-in hard drive, SmartMediaCard (SMC), Secure Digital (SD) card, or FlashCard. Furthermore, the memory can include both internal and external storage units of the terminal device. The memory is used to store the program code and other programs and data required by the terminal device. The memory can also be used to temporarily store data that has been output or will be output.

[0064] The computer storage medium and terminal device described above are created based on the referral decision method described above. Their technical functions and beneficial effects will not be elaborated here. The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0065] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of this invention patent should be determined by the appended claims.

Claims

1. A referral decision-making method based on multimodal fusion, characterized in that, include: S1: Acquire multimodal data in referral scenarios; S2: Encode the features of the multimodal data separately to obtain the feature vectors of the corresponding modes; and fuse the feature vectors to obtain the multimodal fused features; S3: Based on multimodal fusion features, predict several second-layer sub-dimension fit values; aggregate the several second-layer sub-dimension fit values ​​to obtain several first-layer major category fit values; aggregate the first-layer major category fit values ​​to obtain a comprehensive fit value; S4: Generate referral decision results based on the comprehensive fit value.

2. The referral decision-making method according to claim 1, characterized in that, Multimodal data, including semantic text modality, quantified numerical modality, medical consortium graph structure modality, and referral policy coding modality, is encoded to obtain text feature vectors, numerical feature vectors, graph attention vectors, and ontology path vectors. The steps for fusing these feature vectors to obtain multimodal fused features include: The text feature vector and the numerical feature vector are concatenated and input into the gating weight calculation unit; the learnable weight matrix and the bias term are linearly transformed, and then the activation function is applied to generate adaptive gating weights. Based on adaptive gating weights, the text feature vector and the numerical feature vector are weighted and fused to obtain the gating weighted text-numerical fusion feature; The graph attention vector is linearly transformed by a learnable projection matrix and mapped to a feature space consistent with the text-numerical fusion feature. Then, it is added to the text-numerical fusion feature to obtain the cross-type joint feature. By concatenating cross-type joint features with ontology path vectors and applying shared linear transformation and ReLU nonlinear activation, multimodal fusion features are obtained.

3. The referral decision-making method according to claim 2, characterized in that, The semantic text modality is encoded by the following steps: word embedding is followed by bidirectional LSTM to obtain the hidden state sequence, and then time-dimensional attention pooling is used to obtain the semantic feature vector. The quantized numerical modes are feature-encoded, specifically by extracting numerical feature vectors through a combination of multilayer perceptron and segmented activation.

4. The referral decision-making method according to claim 2, characterized in that, Feature encoding of the medical consortium diagram structure modality is performed as follows: First, by stacking multi-layer graph attention networks, multi-level attention weighted aggregation is performed on the institutional node features of heterogeneous subgraphs in the medical consortium graph structure; Then, a graph-level mean-maximum concatenation operation is performed on the aggregated node features. The global mean and global maximum values ​​of the node features are calculated and concatenated to merge the global statistical features and extreme value features of the subgraph. Finally, the concatenated features are mapped into a fixed-dimensional graph attention vector through a linear projection layer.

5. The referral decision-making method according to claim 2, characterized in that, Feature encoding is performed on the referral policy coding modality. Specifically, the policy code is embedded into a pre-set fixed vocabulary to generate an ontology path vector.

6. The referral decision-making method according to claim 1, characterized in that, Step S3: Use the preset formulas (1)-(2) to aggregate and obtain the first-level category fit value and overall fit value: （1）； （2）； in, This represents the number of sub-dimensions in the second layer. The number of items in the first-level category. For the first The first of the major categories Weights of each sub-dimension For the first The second-level sub-dimension fit value For the first The set of sub-dimensions contained in each major category for The first-level category fit value For the first Weights of each major category, This is the overall fit value; among which, , Pre-set.

7. The referral decision-making method according to claim 6, characterized in that, Step S4: Using formula (3), determine the referral priority level based on the comprehensive fit value: （3）； in, For the first Thresholds for each level, For the number of levels, Priority for referral; Alternatively, construct an integer linear programming model with a set of candidate receiving institutions. Institutions in China To define the decision object, introduce 0-1 decision variables. , Indicates that the organization Selected as the sole primary recommender recipient; The optimization objective is to maximize the weighted sum of the target score and the corresponding decision variables, based on the target score determined by the second-level sub-dimensional fitness value, the comprehensive fitness value, and the institutional characteristics. Set constraints: the sum of decision variables for all institutions is 1, the decision variable for each institution is no greater than its corresponding feasibility indicator variable, and all decision variables take the value of 0 or 1; the optimal solution of the integer linear programming model is used as the final primary recommending institution.

8. The referral decision-making method according to any one of claims 1 to 7, characterized in that, A referral decision model is constructed and trained, comprising: an input layer, a feature extraction layer, a hierarchical prediction layer, and an output layer, which are used to execute steps S1-S4 respectively. The training process of the referral decision model employs a multi-level loss function, including: Based on the predicted and labeled values ​​of the fitness scores of the second-level sub-dimensions, the mean squared error loss is constructed as the second-level sub-dimension loss; based on the predicted and labeled values ​​of the fitness scores of the first-level major categories, the L2 loss is constructed as the first-level major category loss; based on the predicted and labeled values ​​of the comprehensive fitness scores, the mean squared error loss is constructed as the comprehensive fitness score loss. The total loss of the referral decision model is obtained by weighting the second-level sub-dimension loss, the first-level category loss, and the comprehensive fit value loss.

9. The referral decision-making method according to claim 8, characterized in that, The multi-level loss function also includes: constructing a marginal loss as a level consistency loss based on the predicted and labeled values ​​of the comprehensive fitness value; constructing a direction alignment loss between text features and numerical features by constraining the changing trends of the disease course text features and the numerical features related to the urgency of the emergency; constructing an intra-group order relation constraint loss by applying soft order constraints to the sub-dimensions within a certain class; and constructing a feature supervision loss by constructing a feature supervision loss for the dual-path feature output and the label. The total loss of the referral decision model also includes additional weighted summation: rank consistency loss, direction alignment loss, intra-group order constraint loss, and feature supervision loss.

10. A terminal device, characterized in that, It includes a memory and a processor; the memory stores program code that can be executed by the processor; the program code is used to execute the referral decision method according to any one of claims 1 to 9.

Images