Intelligent decision-making method for medical information based on multi-modal data fusion

Abstract

The application relates to the technical field of computer-aided medical decision-making, and particularly discloses a medical information intelligent decision-making assisting method based on multi-modal data fusion. A clinical decision-making intelligent agent is constructed and pre-trained, and at least two self-supervised learning tasks are used to pre-train an encoder so that the encoder learns general feature representation of data. Subsequently, a reinforcement learning framework is introduced to train a decision-making network to simulate a real sequential diagnosis and treatment decision-making process, and structured knowledge of a medical knowledge graph is embedded in training as a constraint condition to ensure medical rationality and safety of decision-making, and the trained intelligent agent is used to process current patient data to generate diagnosis and treatment action decision-making and corresponding decision-making supporting information. Through organic cooperation of self-supervised learning, reinforcement learning and a knowledge graph, the application effectively solves technical obstacles such as great data heterogeneity, poor model interpretability and difficulty in integrating with clinical logic in clinical application, and improves practicability and reliability of an auxiliary decision-making system.

Description

Technical Field

[0001] This invention relates to the field of computer-aided medical decision-making technology, and specifically discloses a medical information intelligent decision-making aid method based on multimodal data fusion. Background Technology

[0002] With the development of medical informatization, massive amounts of multimodal data are generated during clinical diagnosis and treatment, such as medical images, electronic medical records, and various examination reports. Currently, AI-based decision support methods mostly focus on the analysis of single-modal data or the simple splicing and fusion of multi-source information. These methods struggle to fully explore the deep semantic relationships and complementary information between different modalities. More importantly, such models typically rely heavily on large amounts of high-quality, fine-grained manually labeled data for supervised training. However, in the medical field, obtaining such labeled data is costly, time-consuming, and easily influenced by the subjective experience of the annotators. This significantly limits the application and performance of these models in data-scarce or rare diseases.

[0003] Existing decision support models often target static, isolated classification or regression tasks, lacking the ability to simulate the dynamic, sequential decision-making process of clinical diagnosis and treatment. They cannot learn to plan and select the optimal sequence of treatment actions based on the evolution of the patient's condition at different time points. Furthermore, most models operate as black boxes, lacking transparency and interpretability in their decision-making logic, making it difficult to integrate with medical logic rich in prior knowledge, such as clinical guidelines and treatment protocols. This results in the model's output occasionally violating common medical sense or safety boundaries, leading to insufficient trust from clinicians and hindering deep integration with existing clinical workflows.

[0004] Current medical decision support technologies face a series of technical obstacles, including difficulties in effectively integrating and utilizing multimodal data, strong dependence on labeled data, weak sequential decision simulation capabilities, poor model interpretability, and difficulty in embedding medical knowledge. These obstacles render existing systems inadequate in terms of robustness, security, and practicality when dealing with complex and dynamic decision-making scenarios in clinical practice. There is an urgent need for a new method that can comprehensively utilize multimodal information, reduce label dependence, simulate clinical decision-making paths, and ensure that the decision-making process complies with medical standards. Summary of the Invention

[0005] To achieve the above objectives, this application provides the following technical solution: A medical information-based intelligent decision-making aid method based on multimodal data fusion includes: S1: Obtain the current multimodal clinical data of the target patient, input the current multimodal clinical data into the pre-constructed clinical decision-making agent, and obtain the current diagnosis and treatment action decision output by the clinical decision-making agent; S2: Obtain historical multimodal clinical data sequence sets, construct a medical knowledge graph, with medical entities as nodes and medical relationships between entities as edges; S3: Construct a multimodal decision model that includes an encoder and a decision network. The encoder is used to fuse and encode the input multimodal clinical data and output a state representation vector. The decision network is used to output the corresponding diagnosis and treatment action decision and decision confidence based on the state representation vector. S4: The encoder is pre-trained using a self-supervised learning approach. Sample data pairs are extracted from the historical multimodal clinical data sequence set. Different self-supervised learning tasks are designed. Using the sample data pairs and the at least two self-supervised learning tasks, the self-supervised loss is calculated and the parameters of the encoder are updated. S5: After the encoder pre-training is completed, the parameters are frozen, and the structured knowledge of the medical knowledge graph is used as a constraint condition and embedded into the training process of the decision network. Based on the historical multimodal clinical data sequence set, the decision network is trained using a reinforcement learning algorithm to simulate the sequential decision-making process in the diagnosis and treatment sequence. S6: Combine the encoder pre-trained in step S4 with the decision network trained in step S5 to form the clinical decision-making agent, and generate and output corresponding decision support information based on the current diagnosis and treatment action decision.

[0006] Furthermore, the encoder of the multimodal decision model constructed in step S3 specifically includes: The text feature extraction submodule is used to process medical text data and generate text feature vectors; The image feature extraction submodule is used to process medical image data and generate image feature vectors. The multimodal fusion submodule is used to receive the text feature vector and the image feature vector, perform information interaction and fusion through a cross-attention mechanism, and output the state representation vector.

[0007] Furthermore, the pre-training of the encoder using a self-supervised learning method described in step S4 specifically includes the following sub-steps: S41: Randomly sample clinical status nodes from the historical multimodal clinical data sequence set, and associate each node with its corresponding multimodal clinical data as a basic sample set; S42: Perform the first self-supervised learning task, carry out the multimodal contrast learning task, and construct the text positive sample view and the image positive sample view of the same patient by performing different data augmentation operations on the medical text data and medical image data of the same patient. Treat the multimodal data of different patient samples as negative samples, calculate the mutual information maximization loss between the text positive sample view and the image positive sample view, and increase the distance with all negative sample views in the feature space. S43: Perform the second self-supervised learning task, carry out cross-modal masking reconstruction task, randomly mask some key entity words in medical text data or mask local image regions in medical image data, input the masked incomplete data into the encoder, and the attached lightweight reconstruction head predicts the content of the masked text entities or masked image regions based on the state representation vector generated by the encoder, and calculates the reconstruction loss. S44: Perform the third self-supervised learning task, and carry out the temporal consistency prediction task. For the continuous clinical state node pairs in the historical clinical diagnosis and treatment sequence, input the associated multimodal clinical data into the encoder to obtain the corresponding state representation vector. Through the additional lightweight temporal prediction head, based on the previous state representation vector, predict the change direction or key features of the subsequent state representation vector and calculate the prediction loss. S45: The mutual information maximization loss corresponding to the multimodal contrastive learning task, the reconstruction loss corresponding to the cross-modal reconstruction task, and the prediction loss corresponding to the temporal consistency prediction task are weighted and summed to obtain the total self-supervised loss. S46: Using the backpropagation algorithm, update the encoder parameters according to the total self-supervised loss, and iteratively execute steps S41 to S45 until the encoder performance converges on the validation set.

[0008] Further, step S45 includes: Assign a first weight coefficient to the mutual information maximization loss, assign a second weight coefficient to the reconstruction loss, and assign a third weight coefficient to the prediction loss; The values ​​of the first weight coefficient, the second weight coefficient, and the third weight coefficient are dynamically adjusted based on the contribution of each supervised learning task to the performance improvement of the downstream decision task on the validation set. The total self-supervised loss obtained after weighted summation is used to update the encoder parameters.

[0009] Furthermore, the step S5, which involves embedding the structured knowledge of the medical knowledge graph as a constraint into the training process of the decision network, specifically includes: The entities and relationships in the medical knowledge graph are vectorized to obtain the knowledge graph embedding vector. During the reinforcement learning training process, a knowledge consistency reward is defined. When the treatment action decision output by the decision network for the current state representation vector is semantically matched with the reasonable treatment suggestion related to the current state based on the knowledge graph, a positive reward is given. When the treatment action decision conflicts with medical knowledge in the knowledge graph that is clearly indicated as taboo or contradictory, a negative reward is given. The knowledge consistency reward item and the environmental reward item reflecting the effect of patient state transition together constitute the reward signal in step S5.

[0010] Furthermore, the method for determining semantic matching between the reasonable treatment suggestions inferred from the knowledge graph and associated with the current state includes: Based on the current state representation vector, retrieve the most relevant medical entities in the medical knowledge graph; By traversing the relationship edges connected to these medical entities in the knowledge graph, we can find the set of recommended and actionable diagnostic and treatment entities in the graph. Map the diagnostic and treatment decisions output by the decision network to the semantic space where the set of diagnostic and treatment entities resides; If the mapped treatment action decision entity belongs to the recommended set of treatment action entities, it is determined to be a match.

[0011] Furthermore, the step S5, which involves training the decision network using a reinforcement learning algorithm and employing a proximal policy optimization algorithm, includes: During training, an experience replay buffer is maintained to store the quadruples of experience data consisting of the state representation vector obtained by the encoder processing historical state data, the diagnosis and treatment action decision output by the decision network, the reward signal obtained after execution, and the state representation vector of the next state after processing by the encoder. The data in the experience replay buffer is used to sample mini-batch samples for calculating the policy gradient; A constraint term for the range of policy changes is introduced into the objective function of policy update to ensure the stability of the training process.

[0012] Furthermore, the sequential decision-making process in the simulated diagnosis and treatment sequence described in step S5 specifically includes the following detailed steps: S51: Initialize the reinforcement learning environment, which is based on the historical multimodal clinical data sequence set and uses the encoder that has been pre-trained and whose parameters are frozen in step S4 as the environment state generator. S52: For historical clinical diagnosis and treatment sequences, starting from the initial clinical state node, input the multimodal clinical data associated with the node into the encoder to obtain the initial state representation vector s0; S53: The decision network outputs the diagnosis and treatment action decision at and its probability distribution based on the current state representation vector st; S54: Execute the diagnosis and treatment action decision at in the simulation environment. Based on the real record of the historical diagnosis and treatment sequence, determine the next clinical state node that the patient will enter after executing at, and input the multimodal clinical data associated with this node into the encoder to obtain the next state representation vector s(t+1). S55: Calculate the immediate reward rt according to the predefined reward function; the reward function rt consists of: an environmental reward r_env based on the degree of improvement of the patient's physiological indicators, and a knowledge consistency reward r_know according to claim 5, i.e., rt = λ1 * r_env + λ2 * r_know, where λ1 and λ2 are balance coefficients; S56: Store the empirical data (st, at, rt, s(t+1)) into the empirical replay buffer; S57: Determine whether the current clinical diagnosis and treatment sequence has ended or whether the preset maximum decision step size has been reached; if it has not ended, let t = t+1, transition the state to s(t+1), and return to step S53; if it has ended, sample a small batch of experience data from the experience replay buffer. S58: Using the sampled small batch of empirical data, calculate the estimate of the advantage function and the loss of the value function; S59: Based on the objective function of the near-end policy optimization algorithm, combined with the estimated value of the advantage function and the policy change constraint, calculate the policy gradient and update the parameters of the decision network; S510: Repeat steps S52 to S59, traversing multiple sequences in the historical clinical diagnosis and treatment data sequence set until the average cumulative reward of the decision network's policy on the validation sequence set converges.

[0013] Furthermore, the clinical decision-making agent, while outputting the current diagnostic and treatment action decision, also outputs a corresponding explanation of the decision basis. The method for generating the explanation of the decision basis includes: Based on the state representation vector generated by the encoder, the attention mechanism is used to backtrack to the specific part of the input multimodal clinical data that contributes the most to the current decision; Extract matching information between the decision-making process of the decision network and the most relevant paths in the medical knowledge graph; The key information from the retrospective input data is integrated with the knowledge graph matching information using natural language to form a readable text that informs decision-making.

[0014] Furthermore, before inputting the current multimodal clinical data into the pre-built clinical decision-making agent, the following steps are also included: The current multimodal clinical data is subjected to a standardized format check to ensure that its data structure is consistent with the input format expected by the pre-trained encoder; If data is missing or formatted incorrectly, a data quality warning will be triggered, and the user will have to wait for manual confirmation or supplementation before the input operation can be performed.

[0015] This invention relates to the field of computer-aided medical decision-making technology, specifically disclosing an intelligent decision-making aid method for medical information based on multimodal data fusion. A clinical decision-making agent is constructed and pre-trained. The encoder is pre-trained using at least two self-supervised learning tasks to learn the general feature representations of the data. Subsequently, a reinforcement learning framework is introduced to train the decision network to simulate a real sequential diagnosis and treatment decision-making process. Structured knowledge from a medical knowledge graph is embedded as a constraint during training to ensure the medical rationality and safety of the decisions. The trained agent processes current patient data to generate diagnosis and treatment action decisions and corresponding decision support information. This invention effectively solves technical obstacles in clinical applications, such as high data heterogeneity, poor model interpretability, and difficulty in integrating with clinical logic, through the organic synergy of self-supervised learning, reinforcement learning, and knowledge graphs, thereby improving the practicality and reliability of the decision-making aid system. Attached Figure Description

[0016] Figure 1 A flowchart illustrating the workflow of a medical information intelligent decision-making aid method based on multimodal data fusion, as claimed in an embodiment of the present invention. Figure 2 The second flowchart is shown for a medical information intelligent decision-making aid method based on multimodal data fusion, as claimed in an embodiment of the present invention. Figure 3 The third flowchart is a medical information intelligent decision-making aid method based on multimodal data fusion, which is claimed in the embodiments of the present invention. Detailed Implementation

[0017] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0018] The terms "first," "second," and "third" in this application are for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first," "second," or "third" may explicitly or implicitly include at least one of those features. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified. All directional indications in the embodiments of this application, such as up, down, left, right, front, back, etc., are only used to explain the relative positional relationships and movements between components in a specific orientation as shown in the accompanying drawings. If the specific orientation changes, the directional indications will change accordingly. Furthermore, the terms "including" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices.

[0019] References to embodiments herein mean that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0020] According to the first embodiment of the present invention, referring to Figure 1 This invention claims protection for a medical information intelligent decision-making aid method based on multimodal data fusion, comprising: S1: Obtain the current multimodal clinical data of the target patient, input the current multimodal clinical data into the pre-constructed clinical decision-making agent, and obtain the current diagnosis and treatment action decision output by the clinical decision-making agent; S2: Obtain historical multimodal clinical data sequence sets, construct a medical knowledge graph, with medical entities as nodes and medical relationships between entities as edges; S3: Construct a multimodal decision model that includes an encoder and a decision network. The encoder is used to fuse and encode the input multimodal clinical data and output a state representation vector. The decision network is used to output the corresponding diagnosis and treatment action decision and decision confidence based on the state representation vector. S4: The encoder is pre-trained using a self-supervised learning approach. Sample data pairs are extracted from the historical multimodal clinical data sequence set. Different self-supervised learning tasks are designed. Using the sample data pairs and the at least two self-supervised learning tasks, the self-supervised loss is calculated and the parameters of the encoder are updated. S5: After the encoder pre-training is completed, the parameters are frozen, and the structured knowledge of the medical knowledge graph is used as a constraint condition and embedded into the training process of the decision network. Based on the historical multimodal clinical data sequence set, the decision network is trained using a reinforcement learning algorithm to simulate the sequential decision-making process in the diagnosis and treatment sequence. S6: Combine the encoder pre-trained in step S4 with the decision network trained in step S5 to form the clinical decision-making agent, and generate and output corresponding decision support information based on the current diagnosis and treatment action decision.

[0021] In this embodiment, S1 acquires a historical multimodal clinical data sequence set, which contains multiple anonymized historical clinical diagnosis and treatment sequences. Each historical clinical diagnosis and treatment sequence consists of multiple clinical state nodes arranged in chronological order. Each clinical state node is associated with the patient's multimodal clinical data at that moment and the subsequent diagnosis and treatment actions performed. The diagnosis and treatment actions include one or more of the following: ordering specific examinations, initiating specific medications, performing specific surgical procedures, and recommending observation and follow-up. In S2, a medical knowledge graph is constructed, with medical entities as nodes and medical relationships between entities as edges. The medical entities cover diseases, symptoms, signs, drugs, surgeries, and examination and testing items. The medical relationships include etiology, clinical manifestations, examination and diagnosis, treatment methods, drug contraindications, and disease stages. In S3, a multimodal decision model is constructed, which includes an encoder and a decision network. The encoder is used to fuse and encode the input multimodal clinical data and output a fixed-dimensional state representation vector. The decision network is a policy network with a multi-layer fully connected structure, which is used to receive the state representation vector, perform internal calculations, output the probability distribution corresponding to different optional treatment actions, take the action with the highest probability as the current treatment action decision output, and output a decision confidence value between 0 and 1. In S4, a self-supervised learning approach is used to pre-train the encoder: sample data pairs that do not rely on manual labeling are extracted from the historical multimodal clinical data sequence set, at least two different self-supervised learning tasks are designed, and the self-supervised loss is calculated and the parameters of the encoder are updated using the sample data pairs and the at least two self-supervised learning tasks, so that the encoder learns a general feature representation of multimodal clinical data to overcome the deficiency of insufficient labeled data. In step S5, after the encoder pre-training is completed, its parameters are frozen, and the structured knowledge of the medical knowledge graph is used as a constraint and embedded into the training process of the decision network. Based on the historical multimodal clinical data sequence set, a reinforcement learning algorithm is used to train the decision network to simulate the sequential decision-making process in the diagnosis and treatment sequence. In each step of the reinforcement learning training, a reward signal is calculated based on the current state representation vector, the diagnosis and treatment action decision output by the decision network, the next clinical state to which the decision is transferred after execution, and the prior constraints provided by the medical knowledge graph. The parameters of the decision network are updated by optimizing the cumulative reward. In S6, the encoder pre-trained in step S4 is combined with the decision network trained in step S5 to form the clinical decision-making intelligent agent. Based on the current diagnosis and treatment decision, corresponding decision support information is generated and output; the decision support information includes the specific content of the decision, the confidence level, and a summary of key decision evidence generated based on the activation within the decision network.

[0022] Furthermore, the encoder of the multimodal decision model constructed in step S3 specifically includes: The text feature extraction submodule is used to process medical text data and generate text feature vectors; The image feature extraction submodule is used to process medical image data and generate image feature vectors. The multimodal fusion submodule is used to receive the text feature vector and the image feature vector, perform information interaction and fusion through a cross-attention mechanism, and output the state representation vector.

[0023] In this embodiment, the text feature extraction submodule is used to process medical text data and generate text feature vectors. The submodule first segments the text into words, converts the words into vector sequences by looking up a pre-trained medical word vector table, then uses a bidirectional long short-term memory network or a Transformer encoder layer to perform context encoding on the vector sequences, and finally obtains a comprehensive text feature vector through pooling operations. The image feature extraction submodule is used to process medical image data and generate image feature vectors. This submodule uses a pre-trained convolutional neural network as the backbone network, receives medical image data as input, extracts its multi-level visual feature maps, and finally performs global average pooling on the feature maps to obtain a compact image feature vector. The multimodal fusion submodule receives the text feature vector and the image feature vector, performs information interaction and fusion through a cross-attention mechanism, and outputs the state representation vector. The fusion submodule first projects the text feature vector and the image feature vector to a common feature space, then calculates the attention weight of the text feature to the image feature and the attention weight of the image feature to the text feature, performs weighted aggregation of the original features according to the weights, concatenates the two aggregated feature vectors, and performs dimensionality reduction and integration through a fully connected layer, finally outputting the state representation vector.

[0024] Furthermore, referring to Figure 2 The pre-training of the encoder using a self-supervised learning method described in step S4 specifically includes the following sub-steps: S41: Randomly sample clinical status nodes from the historical multimodal clinical data sequence set, and associate each node with its corresponding multimodal clinical data as the basic sample set; S42: Perform the first self-supervised learning task, carry out the multimodal contrast learning task, and construct the text positive sample view and the image positive sample view of the same patient by performing different data augmentation operations on the medical text data and medical image data of the same patient. Treat the multimodal data of different patient samples as negative samples, calculate the mutual information maximization loss between the text positive sample view and the image positive sample view, and increase the distance with all negative sample views in the feature space. S43: Perform the second self-supervised learning task, carry out cross-modal masking reconstruction task, randomly mask some key entity words in medical text data or mask local image regions in medical image data, input the masked incomplete data into the encoder, and the attached lightweight reconstruction head predicts the content of the masked text entities or masked image regions based on the state representation vector generated by the encoder, and calculates the reconstruction loss. S44: Perform the third self-supervised learning task, and carry out the temporal consistency prediction task. For the continuous clinical state node pairs in the historical clinical diagnosis and treatment sequence, input the associated multimodal clinical data into the encoder to obtain the corresponding state representation vector. Through the additional lightweight temporal prediction head, based on the previous state representation vector, predict the change direction or key features of the subsequent state representation vector and calculate the prediction loss. S45: The total self-supervised loss is obtained by weighting and summing the mutual information maximization loss corresponding to the multimodal contrastive learning task, the reconstruction loss corresponding to the cross-modal reconstruction task, and the prediction loss corresponding to the temporal consistency prediction task. S46: Using the backpropagation algorithm, update the encoder parameters based on the total self-supervised loss, and iteratively execute steps S41 to S45 until the encoder performance converges on the validation set.

[0025] In this embodiment, in S1, a batch of clinical status nodes are randomly sampled from the historical multimodal clinical data sequence set, and each node is associated with its corresponding multimodal clinical data as a basic sample set; the sampling process ensures the diversity of samples in terms of disease type and disease stage. In S42, the first self-supervised learning task—multimodal contrastive learning—is executed: For each sample in the base sample set, different data augmentation operations are performed on the medical text data and medical image data of the same patient to construct a text positive sample view and an image positive sample view for that sample, respectively. The text data augmentation operations include synonym replacement, random occlusion of some words, and sentence order transposition; the image data augmentation operations include random cropping, rotation, brightness and contrast adjustment, and addition of Gaussian noise. The multimodal data of different patient samples are regarded as negative samples. The mutual information maximization loss between the text positive sample view and the image positive sample view is calculated. Specifically, after mapping the feature vectors of the two positive sample views in a projection head network, their similarity is calculated and maximized, while increasing the similarity distance between them and the projected feature vectors of all negative sample views. This distance is achieved by calculating the contrastive loss function. In S43, the second self-supervised learning task—cross-modal masking reconstruction—is performed: For each sample in the base sample set, some key entity words in its medical text data or local image regions in its medical image data are randomly masked; the masking strategy is as follows: for text, after identifying medical entities, a certain proportion is randomly selected for masking; for images, a rectangular region is randomly selected and its pixel values ​​are set to zero; the incomplete data after masking is input into the encoder, and an additional lightweight reconstruction head predicts the content of the masked text entities or the masked image regions based on the state representation vector generated by the encoder, and calculates the reconstruction loss; the text reconstruction loss is the cross-entropy loss between the predicted words and the original words, and the image reconstruction loss is the mean square error loss between the predicted pixel region and the original pixel region; In S44, the third self-supervised learning task—temporal consistency prediction—is performed: for consecutive clinical state node pairs extracted from the historical clinical diagnosis and treatment sequence, their associated multimodal clinical data are input into the encoder to obtain corresponding state representation vectors; through an additional lightweight temporal prediction head, based on the preceding state representation vector, the direction of change or key features of the subsequent state representation vector is predicted, and the prediction loss is calculated; the direction of change is represented by the difference vector between the subsequent state representation vector and the preceding state representation vector; the key features are represented by labels automatically extracted from the multimodal data of the subsequent states by an auxiliary classifier; In S45, the mutual information maximization loss corresponding to the multimodal contrastive learning task, the reconstruction loss corresponding to the cross-modal mask reconstruction task, and the prediction loss corresponding to the temporal consistency prediction task are weighted and summed to obtain the total self-supervised loss; the weights are set to be equal in the early stage of training, and then fine-tuned according to the smoothness of the loss decrease of each task. In step S46, the backpropagation algorithm is used to update the encoder parameters based on the total self-supervised loss, and steps S41 to S45 are executed iteratively until the encoder performance converges on an independent validation set. The convergence criterion is that the total self-supervised loss no longer decreases significantly over multiple consecutive training epochs.

[0026] Further, step S45 includes: Assign a first weight coefficient to the mutual information maximization loss, assign a second weight coefficient to the reconstruction loss, and assign a third weight coefficient to the prediction loss; The values ​​of the first weight coefficient, the second weight coefficient, and the third weight coefficient are dynamically adjusted based on the contribution of each supervised learning task to the performance improvement of the downstream decision task on the validation set. The total self-supervised loss obtained after weighted summation is used to update the encoder parameters.

[0027] In this embodiment, a first weight coefficient is assigned to the mutual information maximization loss, a second weight coefficient is assigned to the reconstruction loss, and a third weight coefficient is assigned to the prediction loss. In the initial stage, the first weight coefficient, the second weight coefficient, and the third weight coefficient are all set to the same value so that each task has equal influence in the early stage of training. The values ​​of the first, second, and third weight coefficients are dynamically adjusted based on the contribution of each supervised learning task to the performance improvement of the downstream decision task on the validation set. The adjustment strategy is as follows: After each round of pre-training iteration, a fixed small validation set is used, which contains a small number of samples with diagnostic action labels; a small decision network with the same structure is initialized using encoder parameters obtained only from the loss of a single self-supervised task in the current round, and fine-tuned on the validation set with fixed steps; then the initial decision accuracy of each fine-tuned network on the validation set is evaluated; based on the ratio of the initial decision accuracy of each task to the average accuracy of the three tasks, the weight coefficients of the corresponding tasks in the next round are slightly normalized and scaled; for example, if the accuracy of a task is higher than the average, its weight is slightly increased in the next round, and the increase is proportional to the higher ratio, but there is an upper limit to prevent a certain task from completely dominating. The weight coefficients updated according to the above dynamic adjustment strategy are multiplied by the corresponding task loss and then summed to obtain the total self-supervised loss used to update the encoder parameters in this iteration.

[0028] Furthermore, the step S5, which involves embedding the structured knowledge of the medical knowledge graph as a constraint into the training process of the decision network, specifically includes: The entities and relationships in the medical knowledge graph are vectorized to obtain the knowledge graph embedding vector. During the reinforcement learning training process, a knowledge consistency reward is defined. When the treatment action decision output by the decision network for the current state representation vector is semantically matched with the reasonable treatment suggestion related to the current state based on the knowledge graph, a positive reward is given. When the treatment action decision conflicts with medical knowledge in the knowledge graph that is clearly indicated as taboo or contradictory, a negative reward is given. In this embodiment, knowledge graph embedding technology is used to map the entities and relations in the medical knowledge graph to a continuous vector space to obtain knowledge graph embedding vectors; specifically, TransE or its variants are used to learn the distributed representation of each entity and relation by minimizing the differences between the head entity vector, relation vector and tail entity vector. Secondly, during the reinforcement learning training process, a knowledge consistency reward is defined: when the treatment action decision output by the decision network for the current state representation vector semantically matches the reasonable treatment suggestion inferred from the knowledge graph and associated with the current state, a positive reward is given; the higher the matching degree, the larger the positive reward value; when the treatment action decision conflicts with medical knowledge in the knowledge graph that is explicitly indicated as taboo or contradictory, a strong negative reward is given; the judgment of matching and conflict is achieved by calculating the similarity between the vector representation of the treatment action decision in the semantic space and the related entity vector in the knowledge graph, combined with the logical rules of relational paths. Finally, the knowledge consistency reward item and the environmental reward item reflecting the effect of patient state transition together constitute the reward signal described in step S5. The environmental reward item is calculated based on the degree of improvement or deterioration of key physiological indicators such as laboratory values ​​and symptom scores in the patient's clinical state nodes after the implementation of the treatment action. The two rewards are linearly combined through a pre-set coefficient to form a comprehensive reward signal, which is used to guide the strategy update of the decision network.

[0029] The knowledge consistency reward item and the environmental reward item reflecting the effect of patient state transition together constitute the reward signal in step S5.

[0030] Furthermore, referring to Figure 3 The method for determining whether a reasonable treatment suggestion based on knowledge graph reasoning and associated with the current state is semantically matched includes: Based on the current state representation vector, retrieve the most relevant medical entities in the medical knowledge graph; By traversing the relationship edges connected to these medical entities in the knowledge graph, we can find the set of recommended and actionable diagnostic and treatment entities in the graph. Map the diagnostic and treatment decisions output by the decision network to the semantic space where the set of diagnostic and treatment entities resides; If the mapped treatment action decision entity belongs to the recommended set of treatment action entities, it is determined to be a match.

[0031] In this embodiment, the first step is to map the current state representation vector to the same semantic vector space as the knowledge graph embedding through a trainable mapping layer to obtain a query vector. The second step is to perform an approximate nearest neighbor search in the knowledge graph entity vector set using the query vector, and retrieve the top K medical entities with the highest cosine similarity to it, which are the entities most relevant to the current state. The third step is to start with these K entities and traverse the knowledge graph along positive relationship edges such as indications, recommended treatments, and next steps in a finite number of steps to collect all reachable entities of the type of diagnosis and treatment action, forming a set of candidate entities for diagnosis and treatment actions recommended by the graph. The fourth step is to transform the diagnosis and treatment decisions output by the decision network, which are usually specific codes or names, into corresponding knowledge graph entities through a fixed mapping table, and obtain their vector representations in the knowledge graph embedding space. The fifth step is to calculate the semantic similarity between the decision entity vector and each entity vector in the candidate entity set of the treatment action recommended by the graph; if the highest similarity exceeds the preset matching threshold, it is determined to be a match; in addition, if the decision entity and an entity in the candidate set are directly connected by being equivalent to or having a relationship, there is no need to calculate the similarity, and it is directly determined to be a strong match.

[0032] Furthermore, the step S5, which involves training the decision network using a reinforcement learning algorithm and employing a proximal policy optimization algorithm, includes: During training, an experience replay buffer is maintained to store the quadruples of experience data consisting of the state representation vector obtained by the encoder processing historical state data, the diagnosis and treatment action decision output by the decision network, the reward signal obtained after execution, and the state representation vector of the next state after processing by the encoder. The data in the experience replay buffer is used to sample mini-batch samples for calculating the policy gradient; A constraint term for the range of policy changes is introduced into the objective function of policy update to ensure the stability of the training process.

[0033] Furthermore, the sequential decision-making process in the simulated diagnosis and treatment sequence described in step S5 specifically includes the following detailed steps: S51: Initialize the reinforcement learning environment, which is based on the historical multimodal clinical data sequence set and uses the encoder that has been pre-trained and whose parameters are frozen in step S4 as the environment state generator. S52: For historical clinical diagnosis and treatment sequences, starting from the initial clinical state node, input the multimodal clinical data associated with the node into the encoder to obtain the initial state representation vector s0; S53: The decision network outputs the diagnosis and treatment action decision at and its probability distribution based on the current state representation vector st; S54: Execute the diagnosis and treatment action decision at in the simulation environment. Based on the real record of the historical diagnosis and treatment sequence, determine the next clinical state node that the patient will enter after executing at, and input the multimodal clinical data associated with this node into the encoder to obtain the next state representation vector s(t+1). S55: Calculate the immediate reward rt according to the predefined reward function; the reward function rt consists of: an environmental reward r_env based on the degree of improvement of the patient's physiological indicators, and a knowledge consistency reward r_know according to claim 5, i.e., rt = λ1 * r_env + λ2 * r_know, where λ1 and λ2 are balance coefficients; S56: Store the empirical data (st, at, rt, s(t+1)) into the empirical replay buffer; S57: Determine whether the current clinical diagnosis and treatment sequence has ended or whether the preset maximum decision step size has been reached; if it has not ended, let t = t+1, transition the state to s(t+1), and return to step S53; if it has ended, sample a small batch of experience data from the experience replay buffer. S58: Using the sampled small batch of empirical data, calculate the estimate of the advantage function and the loss of the value function; S59: Based on the objective function of the near-end policy optimization algorithm, combined with the estimated value of the advantage function and the policy change constraint, calculate the policy gradient and update the parameters of the decision network; S510: Repeat steps S52 to S59, traversing multiple sequences in the historical clinical diagnosis and treatment data sequence set until the average cumulative reward of the decision network's policy on the validation sequence set converges.

[0034] In this embodiment, in step S51, a reinforcement learning environment is initialized. The environment is based on the historical multimodal clinical data sequence set and uses the encoder that has been pre-trained and whose parameters are frozen in step S4 as the environment state generator. The environment maintains a pointer that points to the current simulated diagnosis and treatment sequence and the current state node in the sequence. In S52, for a historical clinical diagnosis and treatment sequence, starting from the initial clinical state node, the multimodal clinical data associated with the node is input into the encoder, and the encoder performs forward calculation to obtain the initial state representation vector s0. In S53, the decision network receives the current state representation vector st. Each layer inside it performs linear transformation and nonlinear activation on the input vector. Finally, in the output layer, the probability of each possible treatment action is calculated through the Softmax function. Based on this probability distribution, sampling is performed or the action with the highest probability is directly selected to obtain the treatment action decision at. The probability value π(at|st) when at is selected is recorded. In S54, a diagnosis and treatment action decision (at) is executed in the simulation environment. Based on the real records of historical diagnosis and treatment sequences, the environment determines the next clinical state node that the patient will enter after executing at. If a corresponding subsequent node exists in the historical sequence, it is taken as the next state; if at has not been executed in the history, or the sequence has ended, it is processed according to preset rules, such as entering a general sequence termination state. Then, the multimodal clinical data associated with the next node is input into the encoder to obtain the next state representation vector s(t+1). In S55, an immediate reward rt is calculated based on a predefined reward function. The reward function rt comprises: an environmental reward r_env based on the degree of improvement in the patient's physiological indicators, and a knowledge consistency reward r_know as described in claim 5. The environmental reward r_env is calculated by comparing the differences between the next state and the current state on several preset key quantitative indicators. A positive difference is given as a positive reward, and a negative difference is given as a negative reward. The knowledge consistency reward r_know is calculated according to the method of claim 6. A positive reward is given for a match, and a strong negative reward is given for a conflict. Finally, rt = λ1 * r_env + λ2 * r_know, where λ1 and λ2 are predefined balance coefficients used to adjust the relative importance of the two rewards. In S56, the empirical data (st, at, rt, s(t+1)) and the decision probability π(at|st) are stored together in the empirical replay buffer. In step S57, it is determined whether the current clinical diagnosis and treatment sequence has ended, i.e., reached the historical record endpoint or entered the termination state, or whether the current simulation step count has reached the preset maximum decision step size; if it has not ended, let t = t+1, update the current state to s(t+1), and return to step S53; if it has ended, sample a small batch of experience data from the experience replay buffer according to priority or uniform randomness. S58 uses sampled small-batch empirical data to calculate the advantage function estimate and the value function loss. The advantage function estimate measures the performance of a certain action relative to the average performance in that state, and is calculated by combining time-series difference error with the generalized advantage estimation algorithm. The value function loss estimates the value of the state through the value head of a separate value network or decision network, and calculates the error between its predicted value and the actual return. In S59, the policy gradient is calculated based on the objective function of the near-end policy optimization algorithm, combined with the calculated advantage function estimate and the policy change constraint. The calculation of the policy gradient involves the product of the probability ratio of the old and new policies and the advantage function, and is subject to the constraint. Then, stochastic gradient descent or its variants are used to update the parameters of the decision network based on this policy gradient. In S510, repeat steps S52 to S59, traversing multiple sequences in the historical clinical diagnosis and treatment data sequence set until the average cumulative reward of the decision network's strategy on the validation sequence set no longer increases, or the fluctuation is less than the threshold in multiple consecutive training cycles, then it is determined to be converged.

[0035] Furthermore, the clinical decision-making agent, while outputting the current diagnostic and treatment action decision, also outputs a corresponding explanation of the decision basis. The method for generating the explanation of the decision basis includes: Based on the state representation vector generated by the encoder, the attention mechanism is used to backtrack to the specific part of the input multimodal clinical data that contributes the most to the current decision; Extract matching information between the decision-making process of the decision network and the most relevant paths in the medical knowledge graph; The key information from the retrospective input data is integrated with the knowledge graph matching information using natural language to form a readable text that informs decision-making.

[0036] In this embodiment, during the training process, a first-in-first-out experience replay buffer is maintained to store the state representation vector obtained by the encoder processing historical state data, the diagnosis and treatment action decision output by the decision network including the action identifier and the probability of selecting the action, the reward signal obtained after execution, and the four-tuple experience data consisting of the state representation vector of the next state after processing by the encoder; the buffer has a maximum capacity, and when it is full, the new experience data will overwrite the oldest data. The data in the experience replay buffer is used to sample small batches of samples for calculating policy gradients. During sampling, different priorities can be assigned according to the magnitude of the reward obtained from the experience data, so that experiences with high rewards or high prediction errors are sampled more frequently, thereby improving learning efficiency. A constraint term for the range of policy changes is introduced into the objective function of policy update to ensure the stability of the training process. This constraint term is achieved by calculating the probability ratio of the decision network before and after the parameter update to choose the same action, and limiting the ratio to a range close to 1. If the probability ratio exceeds this range, the policy gradient is pruned to prevent a single update from causing too much change to the policy and avoid training crash.

[0037] Furthermore, before inputting the current multimodal clinical data into the pre-built clinical decision-making agent, the following steps are also included: The current multimodal clinical data is subjected to a standardized format check to ensure that its data structure is consistent with the input format expected by the pre-trained encoder; If data is missing or formatted incorrectly, a data quality warning will be triggered, and the user will have to wait for manual confirmation or supplementation before the input operation can be performed.

[0038] In this embodiment, a data standardization format check process is initiated, which loads a predefined input data pattern configuration file; The configuration file is used to compare the structure of the current multimodal clinical data: check whether the medical text data conforms to the predetermined field structure, such as whether it contains the necessary chief complaint and present medical history paragraphs, and check whether the medical image data conforms to the expected dimensions, number of slices, pixel spacing and numerical range. If the inspection finds missing data, inconsistent field naming, mismatched image dimensions, or abnormal values ​​that exceed the physiological range, a data quality warning will be triggered, which will clearly list the non-compliance items and their specific locations. The system enters a waiting state, presenting data quality prompts to the user through an interactive interface, awaiting manual confirmation or supplementation; the user can choose to ignore minor warnings, manually supplement missing data, or reformat the data; Only after receiving manually confirmed or supplemented data, and after passing the format check again, is the operation of inputting the data into the clinical decision-making agent performed.

[0039] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or indirect coupling or communication connection between apparatuses or units, and may be electrical, mechanical, or other forms.

[0040] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated units described above can be implemented in hardware or as software functional units. The above are merely embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made based on the description and drawings of this application, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

[0041] The specific embodiments of the invention have been described in detail above, but they are only examples, and this application is not limited to the specific embodiments described above. For those skilled in the art, any equivalent modifications or substitutions to the invention are also within the scope of this application. Therefore, all equivalent changes, modifications, and improvements made without departing from the spirit and principles of this application should be covered within the scope of this application.

Claims

1. A medical information intelligent decision-making aid method based on multimodal data fusion, characterized in that, include: S1: Obtain the current multimodal clinical data of the target patient, input the current multimodal clinical data into the pre-constructed clinical decision-making agent, and obtain the current diagnosis and treatment action decision output by the clinical decision-making agent; S2: Obtain historical multimodal clinical data sequence sets, construct a medical knowledge graph, with medical entities as nodes and medical relationships between entities as edges; S3: Construct a multimodal decision model that includes an encoder and a decision network. The encoder is used to fuse and encode the input multimodal clinical data and output a state representation vector. The decision network is used to output the corresponding diagnosis and treatment action decision and decision confidence based on the state representation vector. S4: The encoder is pre-trained using a self-supervised learning approach. Sample data pairs are extracted from the historical multimodal clinical data sequence set. Different self-supervised learning tasks are designed. Using the sample data pairs and the at least two self-supervised learning tasks, the self-supervised loss is calculated and the parameters of the encoder are updated. S5: After the encoder pre-training is completed, the parameters are frozen, and the structured knowledge of the medical knowledge graph is used as a constraint condition and embedded into the training process of the decision network. Based on the historical multimodal clinical data sequence set, the decision network is trained using a reinforcement learning algorithm to simulate the sequential decision-making process in the diagnosis and treatment sequence. S6: Combine the encoder pre-trained in step S4 with the decision network trained in step S5 to form the clinical decision-making agent, and generate and output corresponding decision support information based on the current diagnosis and treatment action decision.

2. The method according to claim 1, characterized in that, The encoder of the multimodal decision model constructed in step S3 specifically includes: The text feature extraction submodule is used to process medical text data and generate text feature vectors; The image feature extraction submodule is used to process medical image data and generate image feature vectors. The multimodal fusion submodule is used to receive the text feature vector and the image feature vector, perform information interaction and fusion through a cross-attention mechanism, and output the state representation vector.

3. The method according to claim 1 or 2, characterized in that, Step S4, which involves pre-training the encoder using a self-supervised learning approach, specifically includes the following sub-steps: S41: Randomly sample clinical status nodes from the historical multimodal clinical data sequence set, and associate each node with its corresponding multimodal clinical data as a basic sample set; S42: Perform the first self-supervised learning task, carry out the multimodal contrast learning task, and construct the text positive sample view and the image positive sample view of the same patient by performing different data augmentation operations on the medical text data and medical image data of the same patient. Treat the multimodal data of different patient samples as negative samples, calculate the mutual information maximization loss between the text positive sample view and the image positive sample view, and increase the distance with all negative sample views in the feature space. S43: Perform the second self-supervised learning task, carry out cross-modal masking reconstruction task, randomly mask some key entity words in medical text data or mask local image regions in medical image data, input the masked incomplete data into the encoder, and the attached lightweight reconstruction head predicts the content of the masked text entities or masked image regions based on the state representation vector generated by the encoder, and calculates the reconstruction loss. S44: Perform the third self-supervised learning task, and carry out the temporal consistency prediction task. For the continuous clinical state node pairs in the historical clinical diagnosis and treatment sequence, input the associated multimodal clinical data into the encoder to obtain the corresponding state representation vector. Through the additional lightweight temporal prediction head, based on the previous state representation vector, predict the change direction or key features of the subsequent state representation vector and calculate the prediction loss. S45: The mutual information maximization loss corresponding to the multimodal contrastive learning task, the reconstruction loss corresponding to the cross-modal reconstruction task, and the prediction loss corresponding to the temporal consistency prediction task are weighted and summed to obtain the total self-supervised loss. S46: Using the backpropagation algorithm, update the encoder parameters according to the total self-supervised loss, and iteratively execute steps S41 to S45 until the encoder performance converges on the validation set.

4. The method according to claim 3, characterized in that, Step S45 includes: Assign a first weight coefficient to the mutual information maximization loss, assign a second weight coefficient to the reconstruction loss, and assign a third weight coefficient to the prediction loss; The values ​​of the first weight coefficient, the second weight coefficient, and the third weight coefficient are dynamically adjusted based on the contribution of each supervised learning task to the performance improvement of the downstream decision task on the validation set. The total self-supervised loss obtained after weighted summation is used to update the encoder parameters.

5. The method according to claim 1, characterized in that, Step S5, which involves embedding the structured knowledge of the medical knowledge graph as a constraint into the training process of the decision network, specifically includes: The entities and relationships in the medical knowledge graph are vectorized to obtain the knowledge graph embedding vector. During the reinforcement learning training process, a knowledge consistency reward is defined. When the treatment action decision output by the decision network for the current state representation vector is semantically matched with the reasonable treatment suggestion related to the current state based on the knowledge graph, a positive reward is given. When the treatment action decision conflicts with medical knowledge in the knowledge graph that is clearly indicated as taboo or contradictory, a negative reward is given. The knowledge consistency reward item and the environmental reward item reflecting the effect of patient state transition together constitute the reward signal in step S5.

6. The method according to claim 5, characterized in that, The method for determining whether a reasonable medical suggestion, derived from knowledge graph reasoning and associated with the current state, is semantically matched includes: Based on the current state representation vector, retrieve the most relevant medical entities in the medical knowledge graph; By traversing the relationship edges connected to these medical entities in the knowledge graph, we can find the set of recommended and actionable diagnostic and treatment entities in the graph. Map the diagnostic and treatment decisions output by the decision network to the semantic space where the set of diagnostic and treatment entities resides; If the mapped treatment action decision entity belongs to the recommended set of treatment action entities, it is determined to be a match.

7. The method according to claim 1, characterized in that, Step S5, which involves training the decision network using a reinforcement learning algorithm and employing a proximal policy optimization algorithm, includes: During training, an experience replay buffer is maintained to store the quadruples of experience data consisting of the state representation vector obtained by the encoder processing historical state data, the diagnosis and treatment action decision output by the decision network, the reward signal obtained after execution, and the state representation vector of the next state after processing by the encoder. The data in the experience replay buffer is used to sample mini-batch samples for calculating the policy gradient; A constraint term for the range of policy changes is introduced into the objective function of policy update to ensure the stability of the training process.

8. The method according to claim 1, characterized in that, The sequential decision-making process in the simulated diagnosis and treatment sequence described in step S5 specifically includes the following detailed steps: S51: Initialize the reinforcement learning environment, which is based on the historical multimodal clinical data sequence set and uses the encoder that has been pre-trained and whose parameters are frozen in step S4 as the environment state generator. S52: For historical clinical diagnosis and treatment sequences, starting from the initial clinical state node, input the multimodal clinical data associated with the node into the encoder to obtain the initial state representation vector s0; S53: The decision network outputs the diagnosis and treatment action decision at and its probability distribution based on the current state representation vector st; S54: Execute the diagnosis and treatment action decision at in the simulation environment. Based on the real record of the historical diagnosis and treatment sequence, determine the next clinical state node that the patient will enter after executing at, and input the multimodal clinical data associated with this node into the encoder to obtain the next state representation vector s(t+1). S55: Calculate the immediate reward rt based on a predefined reward function; The reward function rt is composed of: an environmental reward r_env based on the degree of improvement of the patient's physiological indicators, and a knowledge consistency reward r_know as described in claim 5, i.e., rt = λ1 * r_env + λ2 * r_know, where λ1 and λ2 are balance coefficients; S56: Store the empirical data (st, at, rt, s(t+1)) into the empirical replay buffer; S57: Determine whether the current clinical diagnosis and treatment sequence has ended or whether the preset maximum decision step size has been reached; if it has not ended, let t = t+1, transition the state to s(t+1), and return to step S53; if it has ended, sample a small batch of experience data from the experience replay buffer. S58: Using the sampled small batch of empirical data, calculate the estimate of the advantage function and the loss of the value function; S59: Based on the objective function of the near-end policy optimization algorithm, combined with the estimated value of the advantage function and the policy change constraint, calculate the policy gradient and update the parameters of the decision network; S510: Repeat steps S52 to S59, traversing multiple sequences in the historical clinical diagnosis and treatment data sequence set until the average cumulative reward of the decision network's policy on the validation sequence set converges.

9. The method according to claim 1, characterized in that, The clinical decision-making agent outputs the current diagnosis and treatment action decision, and also outputs a corresponding explanation of the decision basis. The method for generating the explanation of the decision basis includes: Based on the state representation vector generated by the encoder, the attention mechanism is used to backtrack to the specific part of the input multimodal clinical data that contributes the most to the current decision; Extract matching information between the decision-making process of the decision network and the most relevant paths in the medical knowledge graph; The key information from the retrospective input data is integrated with the knowledge graph matching information using natural language to form a readable text that informs decision-making.

10. The method according to claim 1, characterized in that, Before inputting the current multimodal clinical data into the pre-built clinical decision-making agent, the following steps are also included: The current multimodal clinical data is subjected to a standardized format check to ensure that its data structure is consistent with the input format expected by the pre-trained encoder; If data is missing or formatted incorrectly, a data quality warning will be triggered, and the user will have to wait for manual confirmation or supplementation before the input operation can be performed.

Images