A multi-modal medical recommendation method based on a double-layer gating mechanism and related devices

Abstract

The application discloses a multi-modal medical recommendation method based on a double-layer gating mechanism and related devices, and relates to the technical field of artificial intelligence medical treatment. The method obtains text data, numerical data and category data of a patient, and pre-processes the data to obtain a data set; the data set is input into a multi-task learning model for training until a training round is reached or a preset condition is met, a trained multi-task learning model is obtained, the multi-task learning model extracts features from the text data, the numerical data and the category data through a text feature branch, a numerical feature branch and a category feature branch respectively to obtain text features, numerical features and category features; the text features, the numerical features and the category features are simultaneously input into a double-layer gating network for mode-level gating and feature-level gating processing to obtain final feature representations; and the final feature representations are input into a multi-task head network to obtain classification prediction results of a plurality of tasks. The application can provide accurate personalized medical recommendations for patients.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence medical technology, and in particular to a multimodal medical recommendation method and related device based on a two-layer gating mechanism. Background Technology

[0002] In the field of intelligent healthcare, personalized medical recommendations are a key technology for improving the quality of medical services and optimizing the allocation of medical resources. Traditional medical recommendation systems mainly rely on rule engines and expert knowledge bases, which suffer from limited recommendation accuracy, insufficient personalization, and difficulty in handling complex multimodal medical data. With the development of deep learning technology, neural network-based medical recommendation systems have gradually emerged, but existing methods still face challenges in handling multi-task learning, data imbalance, and feature fusion. Existing multimodal medical data fusion methods mainly adopt single-layer gating strategies in their gating mechanisms, that is, information is filtered only from a single granularity (modal level or feature level). These methods have the following limitations:

[0003] (1) Limitations of modal-level single-layer gating: Existing methods can only achieve global weight allocation across modalities. They assign relative importance weights to different modalities (such as text, numerical, and category) through the Softmax function (normalized exponential function), but cannot perform fine-grained screening of the fused feature vectors. This results in noise dimensions participating in subsequent calculations on an equal footing with effective features, affecting model performance.

[0004] (2) Limitations of feature-level single-layer gating: Existing methods can only select at the dimensional level, assigning weights between 0 and 1 to each dimension of the fused feature vector through the Sigmoid function (logic function), but lack the perception of the overall importance of the input modality, which may lead to the dilution of key modality information in the fusion stage, resulting in a decrease in model performance.

[0005] (3) Limitations of single-granularity gating: In multi-task scenarios, single-granularity gating strategies are difficult to meet the following requirements at the same time: (a) Different tasks have different dependencies on different modalities (e.g., the "hypertension risk" task relies more on numerical features, while the "dietary advice" task relies more on text descriptions); (b) Differences in the importance of different feature dimensions under the same task (e.g., key indicators such as blood pressure and blood sugar should be given higher weights); (c) The balance between noise suppression and key signal preservation (it is necessary to suppress modal-level noise and feature-level noise at the same time).

[0006] (4) Other technical problems: Medical data is severely unbalanced, making it difficult to learn from a few categories of samples; there is a lack of effective multimodal feature fusion mechanisms, making it difficult to make full use of multimodal information such as text descriptions and numerical indicators; the recommendation results lack interpretability and are difficult to gain the trust of doctors and patients; there are inter-task interference problems in multi-task learning, with complex mutual influences between different medical prediction tasks. Summary of the Invention

[0007] To address the aforementioned technical challenges, this invention proposes a multimodal medical recommendation method and related apparatus based on a two-layer gating mechanism. This method dynamically integrates textual, numerical, and categorical features through a two-level collaborative screening mechanism (the first layer is a modality-level Softmax gating mechanism for cross-modal global importance allocation, and the second layer is a feature-level Sigmoid gating mechanism for dimensionally fine-grained screening of fused features). This achieves hierarchical information fusion from coarse to fine granular, effectively overcoming the limitations of single-layer gating in multimodal, multi-task scenarios and providing patients with accurate and personalized medical recommendations.

[0008] To achieve the above objectives, the technical solution of the present invention is as follows:

[0009] A multimodal medical recommendation method based on a two-layer gating mechanism includes the following steps:

[0010] The patient's multimodal medical data is acquired and preprocessed to obtain a dataset; the multimodal medical data includes text data, numerical data, and categorical data, and the dataset is divided into a training set and a test set according to a preset ratio;

[0011] Input the dataset into a pre-defined multi-task learning model based on a two-layer gating mechanism for iterative training until the training rounds are reached or the pre-defined conditions are met, and obtain a well-trained multi-task learning model.

[0012] The patient data to be predicted is input into a trained multi-task learning model for prediction, and a personalized medical recommendation plan for the patient is obtained. The personalized medical recommendation plan includes hypertension risk prediction, BMI classification, dietary advice generation, exercise advice generation, rehabilitation advice generation, and health status assessment.

[0013] The multi-task learning model includes a text feature branch, a numerical feature branch, a category feature branch, a two-layer gating network, and a multi-task head network. The processing procedure of the multi-task learning model is as follows: the text feature branch, numerical feature branch, and category feature branch respectively extract features from text data, numerical data, and category data to obtain text features, numerical features, and category features; the text features, numerical features, and category features are simultaneously input into the two-layer gating network for modality-level gating and feature-level gating processing to obtain the final feature representation; the final feature representation is input into the multi-task head network to obtain classification prediction results for several tasks.

[0014] Preferably, the preprocessing includes the following steps:

[0015] The text data includes diagnostic descriptions and symptom descriptions. The text data is standardized using a BERT-Base-Chinese word segmenter to clean up and remove special characters and extra spaces.

[0016] The numerical data includes age, BMI, systolic blood pressure, diastolic blood pressure, blood glucose, heart rate, body temperature, number of medications used, number of chronic diseases, number of surgeries, and risk score. The numerical data is imputed for missing values ​​and standardized using Z-score.

[0017] Preferably, the multi-task learning model includes the following steps during training:

[0018] Determine the evaluation metrics, which include accuracy, F1 score, and area under the curve;

[0019] A hierarchical learning rate strategy is introduced, which uses the training set to iteratively train the multi-task learning model based on the two-layer gating mechanism. The loss value is calculated based on the loss function, and the learning rate is dynamically adjusted according to the loss value through AdamW optimizer and cosine annealing learning rate scheduling. Mixed precision training improves training efficiency and reduces memory usage, gradient clipping prevents gradient explosion, and the performance of the trained model is evaluated using the test set according to the evaluation metrics.

[0020] The loss function is defined as follows:

[0021] ,in This represents the final total loss for a batch of the model. λ is the dynamic confidence scaling factor; 1 + β (1-confidence), where confidence is the average confidence of the current batch predictions, β is the adjustment hyperparameter, N is the total number of samples in the current batch, and C is the total number of categories in the task. The balancing weights for category c; The focus loss weighting factor for category c; This represents the model's predicted probability that sample i belongs to category c. This is the parameter for adjusting focus loss; This is an indicator function.

[0022] Preferably, feature extraction is performed on the text data, numerical data, and category data through the text feature branch, numerical feature branch, and category feature branch respectively to obtain text features, numerical features, and category features, including the following steps:

[0023] The BERT encoder is used to extract features from text data to obtain extracted features. The extracted features are then processed sequentially through a linear transformation layer, a batch normalization layer, a modified linear unit activation function, and a random deactivation layer to obtain text features.

[0024] Numerical features are obtained by processing numerical data through a multilayer perceptron that incorporates a lightweight multi-head attention mechanism.

[0025] The label encoder maps category data to numeric indices, and the numeric indices are converted into dense vectors through an embedding layer. The dense vectors are then processed sequentially through a linear transformation layer, a batch normalization layer, a modified linear unit activation function, and a random deactivation layer to obtain category features.

[0026] Preferably, the processing procedure of the two-layer gating network includes the following:

[0027] Modal gating assigns modal gating coefficients corresponding to task conditions to text features, numerical features, and categorical features using the Softmax function; based on the text features, numerical features, and categorical features, the modal gating coefficients are multiplied element-wise and then concatenated to obtain the fused features;

[0028] Feature-level gating introduces channel attention. After global average pooling of the input fused features, feature-level gating coefficients between 0 and 1 are assigned to each dimension of the fused features through two fully connected layers and a Sigmoid function. The feature-level gating coefficients are then multiplied element-wise with the fused features to obtain the final feature representation.

[0029] Preferably, the multi-task head network includes six independent 2-layer MLP classification heads.

[0030] Preferably, the method further includes data augmentation operations, which include randomly deleting and replacing text features; performing Gaussian perturbation on numerical features; and applying a random masking strategy to class features to mask them as unknown classes with a preset probability.

[0031] Based on the above, the present invention also discloses a multimodal medical recommendation system based on a two-layer gating mechanism, characterized in that it includes:

[0032] The acquisition module is used to acquire patients' multimodal medical data and preprocess it to obtain a dataset; the multimodal medical data includes text data, numerical data, and category data, and the dataset is divided into a training set and a test set according to a preset ratio;

[0033] The training module is used to input the dataset into a preset multi-task learning model based on a two-layer gating mechanism for iterative training until the training rounds are reached or preset conditions are met, thereby obtaining a trained multi-task learning model. The multi-task learning model includes a text feature branch, a numerical feature branch, a category feature branch, a two-layer gating network, and a multi-task head network. The text feature branch, numerical feature branch, and category feature branch respectively extract features from the text data, numerical data, and category data to obtain text features, numerical features, and category features. These text features, numerical features, and category features are simultaneously input into the two-layer gating network for modality-level gating and feature-level gating processing to obtain the final feature representation. The final feature representation is then input into the multi-task head network to obtain classification prediction results for several tasks.

[0034] The prediction module is used to input the patient data to be predicted into a trained multi-task learning model for prediction, and obtain a personalized medical recommendation plan for the patient. The personalized medical recommendation plan includes hypertension risk prediction, BMI classification, dietary advice generation, exercise advice generation, rehabilitation advice generation, and health status assessment.

[0035] Based on the foregoing, the present invention also discloses a computer device, comprising: a memory for storing a computer program; and a processor for executing the computer program to implement any of the methods described above.

[0036] Based on the above, the present invention also discloses a readable storage medium storing a computer program, which, when executed by a processor, implements any of the methods described above.

[0037] Based on the above technical solution, the beneficial effects of the present invention are:

[0038] 1) This invention effectively solves the limitations of single-layer gating: it achieves global importance allocation across modalities through modal-level gating, ensuring that the overall information of key modalities is preserved; it achieves fine-grained screening of fused features through feature-level gating, effectively suppressing noise dimensions and highlighting key signals. The synergistic effect of the two-level gating realizes hierarchical information fusion from coarse-grained to fine-grained.

[0039] 2) This invention significantly improves the performance of multi-task medical recommendation: Under multimodal heterogeneity, the two-layer gating mechanism effectively suppresses information conflicts and highlights key features, achieving stable performance improvements in all six medical recommendation tasks (hypertension risk prediction, BMI classification, dietary suggestion generation, exercise suggestion generation, rehabilitation suggestion generation, and health status assessment). The validation accuracy, F1 score, and AUC value are all significantly better than the original model.

[0040] 3) This invention enhances the interpretability and generalization ability of the model: the dual-level gating mechanism retains modality-level and feature-level weights, which facilitates visualization and interpretation, making the model's decision-making process more interpretable; at the same time, through the two-level collaborative screening mechanism, the model exhibits faster convergence speed and stronger generalization ability during training, providing a reliable guarantee for practical applications.

[0041] 4) This invention provides a new technical approach for the field of intelligent healthcare: The dual-layer gating mechanism proposed in this invention is not only applicable to medical recommendation scenarios but can also be extended to other multimodal and multi-task learning scenarios, providing a new technical path for the in-depth application of artificial intelligence in the medical field. This invention provides a new technical approach for the field of intelligent healthcare, achieving effective fusion of multimodal medical data through a two-level collaborative screening mechanism, possessing significant theoretical value and practical application prospects. At the theoretical level, the dual-layer gating mechanism proposed in this invention enriches the theoretical system of multimodal fusion, providing a new theoretical framework to overcome the limitations of single-layer gating. At the practical application level, this invention can provide patients with more accurate and personalized medical recommendation services, helping to improve the quality of medical services and optimize the allocation of medical resources, providing important technical support for building a smart medical, health, and elderly care service system, and possessing broad application prospects and promotional value. Attached Figure Description

[0042] Figure 1 This is a flowchart illustrating a multimodal medical recommendation method based on a two-layer gating mechanism in one embodiment;

[0043] Figure 2 This is an internal workflow diagram of modal-level gating in one embodiment;

[0044] Figure 3 This is an internal workflow diagram of feature-level gating in one embodiment;

[0045] Figure 4 This is a visualization chart comparing experimental performance before and after one example. Detailed Implementation

[0046] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.

[0047] like Figure 1 As shown, this embodiment provides a multimodal medical recommendation method based on a two-layer gating mechanism, including the following steps:

[0048] Step 1: Acquire and preprocess the patient's multimodal medical data to obtain a dataset; the multimodal medical data includes text data, numerical data, and category data, and the dataset is divided into a training set and a test set according to a preset ratio.

[0049] In this embodiment, Python 3.8 is selected as the programming language, and the hardware environment consists of an Intel Core i7-13620H processor, 32GB DDR4 memory, an NVIDIA RTX 4060 graphics card (8GB VRAM), CUDA version 11.8, and the deep learning framework PyTorch 2.0.1.

[0050] Multimodal medical data from patients was acquired, including textual data (such as diagnostic and symptom descriptions), numerical data (such as age, BMI, systolic blood pressure, diastolic blood pressure, blood glucose, heart rate, body temperature, medication dosage, number of chronic diseases, number of surgeries, risk score, and 11 physiological indicators), and categorical data (such as gender, daily living ability, and rehabilitation history), to construct a dataset. The dataset was divided into 11,529 data points in an 8:2 ratio for training and testing sets. Data from January 2018 to December 2021 was used as the training set (9,223 samples), and data from January 2022 to December 2023 was used as the testing set (2,306 samples). This time-series division method better simulates real-world application scenarios, ensuring the model's effectiveness in time generalization.

[0051] Data preprocessing: Given the high heterogeneity of multimodal medical data (large variations in text sequence length, significant differences in numerical feature dimensions, and uneven distribution of categorical features), multi-step preprocessing is performed on the original data to improve model training efficiency and enhance feature representation capabilities. Details are as follows:

[0052] 1) The text data is standardized using BERT (Bidirectional Encoder Representations from Transformers)-Based-Chinese word segmenter, with a unified sequence length of 128. Sequences shorter than 128 are padded with a special marker [PAD], while sequences longer than 128 are truncated. [CLS] and [SEP] markers are added to the beginning and end of the sequence to ensure the integrity of semantic information. At the same time, the text data is cleaned to remove special characters and extra spaces to maintain the accuracy of medical terms.

[0053] 2) For the 11 numerical data (age, BMI, systolic blood pressure, diastolic blood pressure, blood glucose, heart rate, body temperature, number of medications, number of chronic diseases, number of surgeries, and risk score), missing value imputation (median imputation) and Z-score standardization (standard score standardization) were performed to map them to a standard normal distribution (mean of 0 and standard deviation of 1) to eliminate the influence of dimensions and improve the convergence stability of the model.

[0054] The preprocessed data was augmented using data augmentation strategies based on the PyTorch framework to improve the model's generalization ability and robustness to noise. Random deletion (5% probability) and random replacement (3% probability) were performed on the text data to enhance the model's robustness to text changes. A slight Gaussian perturbation was applied to the numerical data, with a noise standard deviation σ=0.01, introducing moderate randomness while maintaining the data distribution characteristics. A random masking strategy was used for the categorical data, masking the categorical features as unknown categories with a certain probability (5%), improving the model's ability to handle missing information. The augmented data was then input into the improved multi-task learning model for training, and classification prediction results for six tasks were obtained through hyperparameter optimization.

[0055] Step 2: Input the dataset into a pre-defined multi-task learning model based on a two-layer gating mechanism for iterative training until the required number of training rounds or preset conditions are met, thereby obtaining a trained multi-task learning model. The multi-task learning model includes a text feature branch, a numerical feature branch, a category feature branch, a two-layer gating network, and a multi-task head network. The processing procedure of the multi-task learning model is as follows: Features are extracted from text data, numerical data, and category data through the text feature branch, numerical feature branch, and category feature branch, respectively, to obtain text features, numerical features, and category features. These text features, numerical features, and category features are simultaneously input into the two-layer gating network for modality-level gating and feature-level gating processing to obtain the final feature representation. The final feature representation is then input into the multi-task head network to obtain classification prediction results for several tasks.

[0056] In this embodiment, the multi-task learning model includes a text feature branch, a numerical feature branch, a category feature branch, a two-layer gating network (modal-level gating in the first layer and feature-level gating in the second layer), a task-specific fusion layer, and a multi-task head network. The specific configuration is as follows: The text feature branch uses a BERT encoder (BERT-Base-Chinese pre-trained model) with a hidden layer dimension of 768 and a uniform sequence length of 128, compressed to 224 dimensions through a task-specific projection layer; the numerical feature branch uses a 3-layer Multi-Layer Perceptron (MLP) with a hidden layer dimension of [384, 192, 256], equipped with an 8-head multi-head attention mechanism (embedding dimension = 256); the category feature branch has an embedding layer dimension of 12, compressed to 96 dimensions through a 2-layer MLP; the fused feature dimension in the two-layer gated network is 576 (224+256+96), the modality-level gated output dimension is 3 (corresponding to the three modalities of text / numerical / category), and the feature-level gated output dimension is 576 (dimension-wise weights); in the multi-task head network, each of the six tasks uses an independent 2-layer MLP classification head with a hidden layer dimension of [384, number of categories]. The Xavier initialization method is used for the weights and bias matrices to ensure gradient stability in the early stages of training.

[0057] During training, the BERT encoder uses a low learning rate (3 x 10⁻⁶) to avoid destroying its pre-trained language knowledge; modality-level gating and feature-level gating use a medium learning rate (3.9 x 10⁻⁵), while the multi-task head network uses a high learning rate (3.3 x 10⁻⁵) to accelerate its convergence. This hierarchical learning rate strategy protects the BERT pre-trained parameters while allowing the gating network and task heads to quickly adapt to specific tasks. The AdamW optimizer (Adaptive Moment Estimation with Weight Decay), combined with cosine annealing learning rate scheduling, mixed-precision training (AMP), and gradient clipping (maximum norm 1.0), ensures training stability.

[0058] Step 3: Input the patient data to be predicted into the trained multi-task learning model for prediction to obtain a personalized medical recommendation plan for the patient. The personalized medical recommendation plan includes hypertension risk prediction, BMI classification, dietary advice generation, exercise advice generation, rehabilitation advice generation, and health status assessment.

[0059] In this embodiment, to address the specific needs of the medical and health care scenario, independent neural network output heads were designed for six tasks: hypertension risk prediction, BMI classification, dietary suggestion generation, exercise suggestion generation, rehabilitation suggestion generation, and health status assessment. Each output head is a small multilayer perceptron, sharing the fused final feature representation extracted by the preceding modules, but possessing dedicated network parameters (such as 384-dimensional and 192-dimensional hidden layers), ultimately mapping to the category space of its respective task. This parameter isolation mechanism effectively avoids mutual interference between tasks, ensuring the synchronous improvement of prediction accuracy for different medical tasks.

[0060] One embodiment of a multimodal medical recommendation method based on a two-layer gating mechanism also provides a training process for a multi-task learning model, including: determining evaluation metrics, such as accuracy, F1 score, and area under the curve (AUC); introducing a hierarchical learning rate strategy, iteratively training the multi-task learning model based on the two-layer gating mechanism using the training set, calculating the focus loss function value, continuously adjusting the model parameters through the AdamW optimizer based on the focus loss function value, dynamically adjusting the learning rate using cosine annealing learning rate scheduling, improving training efficiency and reducing memory usage through mixed precision training, preventing gradient explosion through gradient pruning, and simultaneously evaluating the performance of the trained model using a test set based on the evaluation metrics.

[0061] In this embodiment, a hierarchical learning rate strategy is introduced. The BERT encoder uses a low learning rate (3 x 10⁻⁶) to avoid destroying the language knowledge acquired during pre-training; the modality-level gating and feature-level gating use a medium learning rate (3.9 x 10⁻⁵), and the multi-task head network uses a high learning rate (3.3 x 10⁻⁵) to accelerate its convergence. This hierarchical learning rate strategy can protect the BERT pre-trained parameters while allowing the gating network and task head to quickly adapt to specific tasks.

[0062] Task weights are dynamically calculated based on the imbalance ratio of the training data: the weight of the i-th task is equal to its base weight multiplied by the ratio of its imbalance ratio to the maximum imbalance ratio among all tasks. This allows for the allocation of larger weights to tasks with higher imbalance levels, balancing the learning difficulty of different tasks. Focus loss addresses the data imbalance problem. Its core idea is to reduce the weight of easily classified samples by using a focus parameter, causing the model to focus more on difficult-to-classify samples. Specifically, the focus loss is determined by the predicted probability, class weights, and the focus parameter, which is set to 0.7 in this embodiment to control the weight allocation between easy and difficult samples.

[0063] To systematically address the prevalent class imbalance problem in medical data, this model designs a composite loss function that integrates multiple weighting mechanisms. Based on cross-entropy loss, this function organically integrates two core techniques: FocalLoss and class weighting, and dynamically adjusts it based on the model's own prediction confidence. This aims to significantly improve the model's ability to discriminate minority class samples and its overall robustness. Specifically, the loss function consists of the following three core components:

[0064] 1) Focal Loss: This component improves upon the classic cross-entropy loss by introducing an adjustable focusing parameter γ. It dynamically reduces the model's focus on "easy-to-classify samples" (i.e., samples with high prediction probabilities), forcing the model to concentrate more on learning "difficult samples" (i.e., samples with low prediction probabilities and prone to misclassification) during training. This effectively alleviates the problem of insufficient learning of minority class features caused by the majority class samples dominating the gradient.

[0065] 2) Class Weighting: This component introduces a weight factor for each class when calculating the loss, which is inversely proportional to the number of valid samples in the training set. Classes with fewer samples are assigned a higher weight. This provides additional loss amplification for minority class samples, directly balancing the contributions of different classes to the total loss at the data distribution level.

[0066] 3) Dynamic weighting of confidence: To further enhance the stability and adaptability of training, the final total loss is dynamically scaled based on the model's average prediction confidence in the current batch. When the model's prediction confidence is low, the loss is appropriately amplified to encourage the model to learn more actively; when the confidence is high, the adjustment range is reduced accordingly to avoid unnecessary oscillations in the later stages of training.

[0067] The complete loss function formula is as follows, which combines all the above mechanisms:

[0068]

[0069] in This represents the final total loss for a batch of the model. λ is the dynamic confidence scaling factor; 1 + β (1-confidence), where confidence is the average confidence of the current batch predictions, β is the adjustment hyperparameter, N is the total number of samples in the current batch, and C is the total number of categories in the task. The balancing weights for category c; The focus loss weight factor for category c is a learnable or preset hyperparameter used to further fine-tune the importance of different categories in the focus loss. This represents the model's predicted probability that sample i belongs to category c. γ is the focus loss adjustment parameter (γ≥0). When γ=0, the loss degenerates into weighted cross-entropy. As γ increases, the model pays more attention to difficult samples. This is an indicator function.

[0070] In one embodiment of a multimodal medical recommendation method based on a two-layer gating mechanism, a specific process is also provided for extracting features from text data, numerical data, and category data through the text feature branch, numerical feature branch, and category feature branch, respectively, to obtain text features, numerical features, and category features. This process includes the following steps:

[0071] In this embodiment, 1) Text Feature Extraction: Text data (such as diagnostic descriptions and symptom descriptions) carries rich semantic information. To effectively capture the deep semantics in the text, a BERT encoder is used to extract features from the text data. These extracted features are then processed sequentially through a linear transformation layer, a batch normalization layer, a Rectified Linear Unit (ReLU) activation function, and a Dropout layer to obtain the text features. During model training, a partial unfreezing strategy is used for the BERT parameters, with only the last few layers and pooling layers being fine-tuned. This allows the model to utilize the general language knowledge learned by the pre-trained model while adapting to the specific context of the medical field, improving the understanding of medical texts while maintaining model stability. Finally, the text modality is encoded into a 224-dimensional feature vector.

[0072] 2) Numerical Feature Extraction: A multilayer perceptron is used to extract high-order statistical information from the numerical features. Considering the potentially complex nonlinear relationships between different physiological indicators (such as blood pressure and heart rate, blood glucose and BMI), a lightweight multi-head self-attention mechanism is introduced on top of the multilayer perceptron. This enables the model to adaptively capture cross-indicator correlations and highlight the importance of key indicators according to task requirements. Finally, the numerical modality is transformed into a 256-dimensional feature vector.

[0073] 3) Category Feature Extraction: Category data includes discrete fields such as gender, daily living ability, and rehabilitation history. First, a label encoder maps the category data (gender, daily living ability, rehabilitation history, etc.) to integer indices. Then, a trainable embedding layer transforms the discrete integer indices into a continuous, dense vector space representation. During training, the embedding layer learns the latent semantic relationships between categories, such as the similarity between different rehabilitation history categories. After concatenating the embedding vectors from multiple fields, a small multilayer perceptron consisting of a linear transformation layer, a batch normalization layer, a modified linear unit activation function, and a random deactivation layer is used to compress the data into a compact 96-dimensional representation, reducing the risk of overfitting caused by sparse category features.

[0074] Through the above processing, the three modalities yield feature representations of 224 dimensions (text), 256 dimensions (numerical), and 96 dimensions (categorical), respectively. Although the original dimensions and semantic spaces of each modality differ, they are unified into a compatible semantic space through dimension alignment and feature compression in this module. Finally, the feature vectors of the three modalities are concatenated to form a 576-dimensional (224+256+96) joint representation, which serves as the input to the subsequent two-layer gating network.

[0075] One embodiment of a multimodal medical recommendation method based on a two-layer gating mechanism also provides a two-layer gating network processing procedure, which includes the following steps: modality-level gating, assigning modality-level gating coefficients corresponding to task conditions to text features, numerical features, and category features using a softmax function; multiplying text features, numerical features, and category features element-wise with the corresponding modality-level gating coefficients and then concatenating them to obtain fused features; feature-level gating introduces channel attention, performing global average pooling on the input fused features, and assigning feature-level gating coefficients between 0 and 1 to each dimension of the fused features through two fully connected layers and a sigmoid function; multiplying the feature-level gating coefficients element-wise with the fused features to obtain the final feature representation.

[0076] In this embodiment, a two-layer gating network is provided to achieve two-level collaborative screening:

[0077] 1) Modal-level gating (first layer, global modality selection, Softmax), see [link / reference] Figure 2 The text, numerical, and categorical representations are concatenated into a joint representation, and then a task-specific gating network predicts the relative importance weights of the three modalities. The concatenated joint representation is as follows: ]

[0078] in These are modal-level gating coefficients (including weights for text, numerical, and categorical modalities). , , ), by weight matrix Task condition vector z and bias Obtained by Softmax normalization; , , These are feature vectors for three modalities: text, numerical, and category; obtained through element-wise multiplication ( After weighting the features of the three modalities, a concatenation operation ([;]) is performed to obtain the fused representation f. The global contribution of the three modalities is adaptively allocated for different tasks / samples (e.g., "hypertension risk" relies more on numerical features).

[0079] 2) Feature-level gating (second layer, dimension-wise fine-grained screening, Sigmoid), see [link / reference] Figure 3 In fusion representation Above, generate dimension-wise weights for each task. :

[0080]

[0081] Where β is the feature-level gate coefficient (dimension-wise weight vector), derived from the weight matrix. , fusion feature f and bias The feature representation is obtained by applying the Sigmoid activation function σ; ⊙ represents element-wise multiplication, where β and f are multiplied element-wise to obtain the final filtered feature representation. Emphasizing key signals and suppressing noise at the dimensional level improves robustness and recall for long-tail / imbalanced tasks.

[0082] Feature-level gating is responsible for fine-grained, dimension-wise feature selection to suppress noise and highlight key signals highly relevant to the task. The fused features output from modality-level gating are used as input, which also receives conditional information from the same task, allowing the feature selection strategy to adaptively adjust to the task.

[0083] Weight Calculation and Selection: A gating network activated by the Sigmoid function is used to generate a weight between 0 and 1 for each dimension in the fused features. A value close to 1 indicates that the feature dimension is preserved and strengthened, while a value close to 0 indicates that the feature dimension is suppressed. For example, in text data features, specific semantic dimensions that are strongly relevant to the current task may receive a high weight of 0.8-0.9, while dimensions that are irrelevant to the task may be suppressed to 0.1-0.2.

[0084] Enhancement Mechanism: To more accurately model the dependencies between channels, this layer can further introduce a channel attention mechanism (such as a squeezed excitation structure, Squeeze-and-Excitation), capture global contextual information through global average pooling, and then learn the correlation between channels through two fully connected layers, ultimately generating better feature weights.

[0085] After 15 training epochs, both the original model without gating and the multi-task learning model based on dual-layer gating (the improved gating model with dual-layer gating) in this application converged and stabilized on the validation set. The final performance of the multi-task learning model based on dual-layer gating on the validation set is as follows: macro-average accuracy 81.90%, macro-average F1 score 81.35%, macro-average area under the curve (AUC) 96.58%, and validation loss 3.0886. Compared to the original model without gating mechanisms, the macro-average accuracy (76.47%), macro-average F1 score (75.83%), macro-average area under the curve (93.71%), and validation loss (3.2764) represent improvements of +5.43, +5.52, and +2.87 percentage points respectively (relative improvements of approximately 7.10%, 7.28%, and 3.06%), and a reduction in validation loss of 0.1878 (approximately 5.73%). The training curves show that the multi-task learning model based on a two-layer gating mechanism significantly outperforms the baseline from the first training epoch, consistently maintaining higher accuracy / F1 score / area under the curve and lower, smoother loss throughout training, demonstrating faster convergence and stronger generalization ability. The overall results demonstrate that the two-layer gating network (modality-level + feature-level) effectively suppresses information conflicts and highlights key features under multimodal heterogeneity conditions. The first-layer modality-level gating achieves global modality importance allocation, while the second-layer feature-level gating enables fine-grained selection of local features. The synergistic effect of the two gating layers significantly improves the overall performance and robustness of the model in joint learning across six tasks. Finally, the raw output of the model is transformed into a probability distribution using a soft-maximum function to obtain the prediction results for each task. Performance comparison and visualization are performed using a performance comparison script, and the visualization results are shown below. Figure 4 .

[0086] Based on the same inventive concept, this application also provides a multimodal medical system for implementing the multimodal medical recommendation method based on a two-layer gating mechanism described above. The solution provided by this system is similar to the implementation described in the above method, and therefore will not be repeated here.

[0087] In one embodiment, a multimodal medical recommendation system based on a two-layer gating mechanism is also provided, comprising:

[0088] The acquisition module is used to acquire patients' multimodal medical data and preprocess it to obtain a dataset; the multimodal medical data includes text data, numerical data, and category data, and the dataset is divided into a training set and a test set according to a preset ratio;

[0089] The training module is used to input the dataset into a preset multi-task learning model based on a two-layer gating mechanism for iterative training until the training rounds are reached or preset conditions are met, thereby obtaining a trained multi-task learning model. The multi-task learning model includes a text feature branch, a numerical feature branch, a category feature branch, a two-layer gating network, and a multi-task head network. The text feature branch, numerical feature branch, and category feature branch respectively extract features from the text data, numerical data, and category data to obtain text features, numerical features, and category features. These text features, numerical features, and category features are simultaneously input into the two-layer gating network for modality-level gating and feature-level gating processing to obtain the final feature representation. The final feature representation is then input into the multi-task head network to obtain classification prediction results for several tasks.

[0090] The prediction module is used to input the patient data to be predicted into a trained multi-task learning model for prediction, and obtain a personalized medical recommendation plan for the patient. The personalized medical recommendation plan includes hypertension risk prediction, BMI classification, dietary advice generation, exercise advice generation, rehabilitation advice generation, and health status assessment.

[0091] In the above embodiments, each module of a multimodal medical recommendation system based on a two-layer gating mechanism can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0092] In one embodiment, a computer device is also provided, comprising: a memory for storing a computer program; and a processor for executing the computer program to implement the steps as described in all the above method embodiments.

[0093] In one embodiment, a readable storage medium is also provided, on which a computer program is stored, which, when executed by a processor, implements the steps as described in all the above method embodiments.

[0094] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, random access memory can take many forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0095] The embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on its differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0096] The above are merely preferred embodiments of the present application and are not intended to limit the embodiments of the present application. For those skilled in the art, the embodiments of the present application can have various modifications and variations. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the embodiments of the present application should be included within the protection scope of the embodiments of the present application.

Claims

1. A multimodal medical recommendation method based on a two-layer gating mechanism, characterized in that, Includes the following steps: The patient's multimodal medical data is acquired and preprocessed to obtain a dataset; the multimodal medical data includes text data, numerical data, and categorical data, and the dataset is divided into a training set and a test set according to a preset ratio; Input the dataset into a pre-defined multi-task learning model based on a two-layer gating mechanism for iterative training until the training rounds are reached or the pre-defined conditions are met, and obtain a well-trained multi-task learning model. The patient data to be predicted is input into a trained multi-task learning model for prediction, and a personalized medical recommendation plan for the patient is obtained. The personalized medical recommendation plan includes hypertension risk prediction, BMI classification, dietary advice generation, exercise advice generation, rehabilitation advice generation, and health status assessment. The multi-task learning model includes a text feature branch, a numerical feature branch, a category feature branch, a two-layer gating network, and a multi-task head network. The processing steps of the multi-task learning model are as follows: The text feature branch, numerical feature branch, and category feature branch respectively extract features from text data, numerical data, and category data to obtain text features, numerical features, and category features; these features are simultaneously input into the two-layer gating network for modality-level gating and feature-level gating processing to obtain the final feature representation; the final feature representation is then input into the multi-task head network to obtain classification prediction results for several tasks. The processing of the two-layer gated network includes the following: modal-level gating, which assigns modal-level gating coefficients corresponding to the task conditions to text features, numerical features, and category features through the Softmax function; and then performing element-wise multiplication of the text features, numerical features, and category features with the corresponding modal-level gating coefficients and then concatenating them to obtain fused features. Feature-level gating introduces channel attention. After global average pooling of the input fused features, feature-level gating coefficients between 0 and 1 are assigned to each dimension of the fused features through two fully connected layers and a Sigmoid function. The feature-level gating coefficients are then multiplied element-wise with the fused features to obtain the final feature representation. The multi-task learning model training process includes the following steps: determining evaluation metrics, including accuracy, F1 score, and area under the curve; introducing a hierarchical learning rate strategy, using the training set to iteratively train the multi-task learning model based on a two-layer gating mechanism, calculating the loss value based on the loss function, and dynamically adjusting the learning rate according to the loss value through the AdamW optimizer and cosine annealing learning rate scheduling, using mixed precision training to improve training efficiency and reduce memory usage, and gradient clipping to prevent gradient explosion; simultaneously, using the test set to evaluate the performance of the trained model according to the evaluation metrics; the loss function is as follows: ,in This represents the final total loss for a batch of the model. This is the dynamic confidence scaling factor; Where confidence is the average confidence level of the current batch predictions, β is the adjustment hyperparameter, N is the total number of samples in the current batch, and C is the total number of categories in the task; The balancing weights for category c; The focus loss weighting factor for category c; This represents the model's predicted probability that sample i belongs to category c. This is the parameter for adjusting focus loss; This is an indicator function.

2. The multimodal medical recommendation method based on a two-layer gating mechanism according to claim 1, characterized in that, The preprocessing includes the following steps: The text data includes diagnostic descriptions and symptom descriptions. The text data is standardized using a BERT-Base-Chinese word segmenter to clean up and remove special characters and extra spaces. The numerical data includes age, BMI, systolic blood pressure, diastolic blood pressure, blood glucose, heart rate, body temperature, number of medications used, number of chronic diseases, number of surgeries, and risk score. The numerical data is imputed for missing values ​​and standardized using Z-score.

3. The multimodal medical recommendation method based on a two-layer gating mechanism according to claim 1, characterized in that, The text feature branch, numerical feature branch, and category feature branch are used to extract features from the text data, numerical data, and category data respectively to obtain text features, numerical features, and category features, including the following steps: The BERT encoder is used to extract features from text data to obtain extracted features. The extracted features are then processed sequentially through a linear transformation layer, a batch normalization layer, a modified linear unit activation function, and a random deactivation layer to obtain text features. Numerical features are obtained by processing numerical data through a multilayer perceptron with a lightweight multi-head attention mechanism. The label encoder maps category data to numeric indices, and the numeric indices are converted into dense vectors through an embedding layer. The dense vectors are then processed sequentially through a linear transformation layer, a batch normalization layer, a modified linear unit activation function, and a random deactivation layer to obtain category features.

4. The multimodal medical recommendation method based on a two-layer gating mechanism according to claim 1, characterized in that, The multi-task head network includes six independent 2-layer MLP classification heads.

5. The multimodal medical recommendation method based on a two-layer gating mechanism according to claim 1, characterized in that, It also includes data augmentation operations, which include random deletion and random replacement of text features; Gaussian perturbation processing of numerical features; and random masking strategy for category features, which masks the category features as unknown categories with a preset probability.

6. A multimodal medical recommendation system based on a two-layer gating mechanism, characterized in that, include: The acquisition module is used to acquire patients' multimodal medical data and preprocess it to obtain a dataset; The multimodal medical data includes text data, numerical data, and categorical data. The dataset is divided into a training set and a test set according to a preset ratio. The training module is used to input the dataset into a preset multi-task learning model based on a two-layer gating mechanism for iterative training until the training rounds are reached or preset conditions are met, thereby obtaining a trained multi-task learning model. The multi-task learning model includes a text feature branch, a numerical feature branch, a category feature branch, a two-layer gating network, and a multi-task head network. The text feature branch, numerical feature branch, and category feature branch respectively extract features from the text data, numerical data, and category data to obtain text features, numerical features, and category features. These text features, numerical features, and category features are simultaneously input into the two-layer gating network for modality-level gating and feature-level gating processing to obtain the final feature representation. The final feature representation is then input into the multi-task head network to obtain classification prediction results for several tasks. The processing steps of the dual-layer gated network include the following: Modal-level gating, which assigns modal-level gating coefficients corresponding to the task conditions to text features, numerical features, and category features using the Softmax function; Multiplying the text features, numerical features, and category features element-wise with the corresponding modal-level gating coefficients and then concatenating them to obtain fused features; Feature-level gating introduces channel attention, which performs global average pooling on the input fused features and assigns feature-level gating coefficients between 0 and 1 to each dimension of the fused features using two fully connected layers and the Sigmoid function; Multiplying the feature-level gating coefficients element-wise with the fused features to obtain the final feature representation. The multi-task learning model training process includes the following steps: determining evaluation metrics, including accuracy, F1 score, and area under the curve; introducing a hierarchical learning rate strategy, using the training set to iteratively train the multi-task learning model based on a two-layer gating mechanism, calculating the loss value based on the loss function, and dynamically adjusting the learning rate according to the loss value through the AdamW optimizer and cosine annealing learning rate scheduling, using mixed precision training to improve training efficiency and reduce memory usage, gradient clipping to prevent gradient explosion, and simultaneously evaluating the performance of the trained model using a test set according to the evaluation metrics; the loss function is as follows: ,in This represents the final total loss for a batch of the model. This is the dynamic confidence scaling factor; Where confidence is the average confidence level of the current batch predictions, β is the adjustment hyperparameter, N is the total number of samples in the current batch, and C is the total number of categories in the task; The balancing weights for category c; The focus loss weighting factor for category c; This represents the model's predicted probability that sample i belongs to category c. This is the parameter for adjusting focus loss; For indicator functions; The prediction module is used to input the patient data to be predicted into a trained multi-task learning model for prediction, and obtain a personalized medical recommendation plan for the patient. The personalized medical recommendation plan includes hypertension risk prediction, BMI classification, dietary advice generation, exercise advice generation, rehabilitation advice generation, and health status assessment.

7. A computer device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the method as described in any one of claims 1 to 5.

8. A readable storage medium, characterized in that, The readable storage medium stores a computer program that, when executed by a processor, implements the method as described in any one of claims 1 to 5.

Images