Multimodal diagnosis and treatment scheme recommendation system based on reinforcement learning

Abstract

The invention discloses a multi-modal diagnosis and treatment scheme recommendation system based on reinforcement learning, and relates to the technical field of artificial intelligence, and the system comprises a data processing module which is used for obtaining multi-modal medical data of a patient and carrying out the preprocessing of the multi-modal medical data to obtain a multi-modal data set; the feature extraction and fusion module is used for performing feature extraction and fusion based on the multi-modal data set to obtain multi-modal features as the current state of the patient; the training set acquisition module is used for constructing a decision network and inputting the current state of the patient to obtain a current diagnosis and treatment scheme, taking the current state of the patient and the current diagnosis and treatment scheme as data groups, and circulating the process to obtain a plurality of data groups to form a training set; the network optimization module is used for optimizing the decision network based on the training set and the core objective function to obtain an optimized decision network; and the optimal scheme output module is used for obtaining the current state of the patient to be recommended and inputting the current state into the optimized decision network to obtain an optimal diagnosis and treatment scheme. And the disease diagnosis accuracy and comprehensiveness are improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and more specifically to a multimodal diagnosis and treatment plan recommendation system based on reinforcement learning. Background Art

[0002] Currently, doctors need to comprehensively analyze medical data from multiple modalities. Traditional medical decision-making systems are typically based on expert rules or simple statistical models. These methods rely on manually defined features and accumulated experience, making it difficult to fully utilize the potential connections between multimodal data. In recent years, deep learning methods have made significant progress in medical data analysis, but they primarily rely on supervised learning and require large amounts of annotated data. However, high-quality annotated data in the medical field is often limited.

[0003] Deep learning methods based on models such as CNN, RNN, and Transformer perform well in single-modal analysis of medical images, text, and numerical data, but still face challenges in multimodal data fusion, such as how to effectively align and utilize data features from different modalities. Existing technologies often use single-modal data information for diagnostic and treatment plan analysis, which cannot fully explore the correlation between data from different modalities and has limitations, leading to low accuracy in disease diagnosis.

[0004] Therefore, how to integrate multimodal medical data to optimize diagnostic and treatment plans, and thereby improve the accuracy and comprehensiveness of disease diagnosis, is an issue that technical personnel in this field urgently need to address. Summary of the Invention

[0005] In view of this, the present invention provides a multimodal diagnosis and treatment plan recommendation system based on reinforcement learning, which optimizes diagnosis and treatment plans by integrating multimodal medical data, thereby improving the accuracy and comprehensiveness of disease diagnosis.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] A multimodal diagnosis and treatment plan recommendation system based on reinforcement learning, including: a data processing module, a feature extraction and fusion module, a training set acquisition module, a network optimization module and an optimal plan output module;

[0008] The data processing module is used to obtain and preprocess the patient's multimodal medical data to obtain a multimodal data set;

[0009] The feature extraction and fusion module is used to extract and fuse features based on the multimodal data set to obtain multimodal features as the patient's current state;

[0010] The training set acquisition module is used to construct a decision network and input the patient's current state to obtain the current diagnosis and treatment plan. Based on the patient's current state and the current diagnosis and treatment plan as a data group, the above process is repeated to obtain multiple data groups to form a training set;

[0011] The network optimization module is used to optimize the decision network based on the training set and the core objective function to obtain an optimized decision network;

[0012] The optimal solution output module is used to obtain the status of the patient to be recommended and input it into the optimized decision network to obtain the optimal diagnosis and treatment solution.

[0013] Preferably, the data processing module includes: a data acquisition unit, a first preprocessing unit, a second preprocessing unit and a labeling processing unit;

[0014] The data acquisition unit is configured to acquire the multimodal medical data of the patient, wherein the multimodal medical data includes: medical report text, diagnosis and treatment dialogue text, and medical images;

[0015] The first preprocessing unit is configured to sequentially perform denoising, resolution adjustment, grayscale standardization, and data enhancement on the medical image to obtain a preprocessed medical image;

[0016] The second preprocessing unit is used to perform text cleaning and data enhancement based on the medical report text and the diagnosis and treatment dialogue text respectively, to obtain a preprocessed medical report text and a preprocessed diagnosis and treatment dialogue text respectively;

[0017] The labeling processing unit is used to assign the same labels to the preprocessed medical images, the preprocessed medical report texts and the preprocessed diagnosis and treatment conversation texts, and then perform standardization processing on them respectively, so as to obtain the final medical images, the final medical report texts and the final diagnosis and treatment conversation texts which together constitute the multimodal dataset.

[0018] Preferably, the feature extraction and fusion module includes: a feature extraction unit and a feature fusion unit;

[0019] The feature extraction unit is used to extract features based on the final medical report text, the final diagnosis and treatment dialogue text and the final medical image, respectively, to obtain medical numerical features, medical text features and medical image features;

[0020] The feature fusion unit is used to perform weighted fusion based on the medical numerical feature, the medical text feature and the medical image feature to obtain the multimodal feature.

[0021] Preferably, the feature extraction unit includes a first feature extraction subunit;

[0022] The first feature extraction subunit is configured to extract all types of numerical data based on the final medical report text, and perform encoding based on the numerical data to obtain numerical information;

[0023] It is also used to perform word segmentation based on the final medical report text and embed word vectors on the segmented text to obtain semantic information of each word, and obtain the medical numerical features based on the corresponding combination of the semantic information and the numerical information.

[0024] Preferably, the feature extraction unit further includes a second feature extraction subunit;

[0025] The second feature extraction subunit is used to perform word segmentation processing and text cleaning in sequence based on the final diagnosis and treatment dialogue text to obtain a cleaned word segmentation text, perform speaker annotation based on the cleaned word segmentation text to obtain annotated word segmentation text, and convert the annotated word segmentation text into a vector representation to obtain the medical text feature.

[0026] Preferably, the feature extraction unit further includes a third feature extraction subunit;

[0027] The third feature extraction subunit is configured to perform image denoising and normalization operations on the 2D medical image in the final medical image to obtain a processed 2D medical image, and perform feature extraction on the processed 2D medical image based on a 2D image encoder to obtain 2D medical image features;

[0028] It is also used to perform image denoising and normalization operations based on the 3D medical image in the final medical image to obtain a processed 3D medical image, and to perform feature extraction on the processed 3D medical image based on a 3D image encoder to obtain 3D medical image features. The 2D medical image features and the 3D medical image features together constitute the medical image features.

[0029] Preferably, the feature fusion unit includes: a weighted feature acquisition subunit and a feature fusion subunit;

[0030] The weighted feature acquisition subunit is used to perform weighted fusion based on the medical numerical features, the medical text features and the medical image features, respectively, to obtain numerical weighted features, text weighted features and image weighted features respectively;

[0031] The feature fusion subunit is used to fuse the numerical weighted features, the text weighted features and the image weighted features to obtain the multimodal features.

[0032] Preferably, the training set acquisition module includes: a data group acquisition unit and a loop execution unit;

[0033] The data group acquisition unit is used to construct a decision network and input the patient's current state to obtain a current diagnosis and treatment plan, obtain a new patient state based on the current diagnosis and treatment plan, obtain a current reward value based on the new patient state and the patient's current state, and use the patient's current state, the current diagnosis and treatment plan, the new patient state, and the current reward value as a data group;

[0034] The loop execution unit is used to loop through the process of obtaining the data group, so as to obtain a plurality of data groups that together constitute the training set.

[0035] Preferably, the current reward value R t for:

[0036] R t =ω1·diagnosis_accuracy+ω2·treatment_effectiveness;

[0037] Among them, ω1 and ω2 represent the first weight factor and the second weight factor respectively, diagnosis_accuracy represents the diagnosis accuracy, and treatment_effectiveness represents the treatment effectiveness.

[0038] Preferably, the core objective function is specifically:

[0039]

[0040]

[0041] Among them, argmax means maximization, E[L PPO (θ)] represents the L PPO Average of (θ), L PPO (θ) represents the objective function, It means taking the average of samples in all time steps t, r t (θ) represents the probability ratio of the new and old strategies, represents the advantage function, and clip(·) represents r t (θ) is restricted to the interval [1-∈, 1+∈], where ∈ represents a hyperparameter.

[0042] It can be seen from the above technical solutions that, compared with the prior art, the present invention discloses a multimodal diagnosis and treatment plan recommendation system based on reinforcement learning, which has the following beneficial effects:

[0043] 1. The present invention adopts special architectures such as numerical encoders (numerical reports) and text encoders (diagnosis and treatment texts) to extract features based on the characteristics of different modal data, accurately obtaining the clinical significance of numerical indicators such as blood pressure and blood sugar, as well as the semantics of text entities such as disease names and symptom descriptions, significantly enhancing the model's ability to understand medical expertise.

[0044] 2. The present invention integrates the numerical data of medical reports, medical texts of diagnosis and treatment dialogues, and multimodal information of 2D / 3D medical images, combines the encoder architecture dedicated to each modality to extract corresponding features, and uses the Transformer architecture for cross-modal feature fusion, fully exploring the correlation between the data of each modality, overcoming the data information limitations of a single modality, and improving the accuracy and comprehensiveness of disease diagnosis.

[0045] 3. The present invention adopts reinforcement learning algorithm training model to optimize diagnosis and treatment strategies. The system can generate personalized diagnosis and treatment plans according to the patient's real-time status (multimodal feature vector), and continuously adjust the strategy through reward function (diagnostic accuracy, treatment effect) to improve treatment effectiveness and clinical adaptability. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0047] Figure 1 This is a schematic diagram of the structure of a multimodal diagnosis and treatment plan recommendation system based on reinforcement learning provided by the present invention.

[0048] Figure 2 This is a structural diagram of the data processing module provided by the present invention.

[0049] Figure 3 This is a structural diagram of the feature extraction unit provided by the present invention. DETAILED DESCRIPTION

[0050] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0051] Example 1

[0052] like Figure 1 As shown, the embodiment of the present invention discloses a multimodal diagnosis and treatment plan recommendation system based on reinforcement learning, including: a data processing module, a feature extraction and fusion module, a training set acquisition module, a network optimization module and an optimal plan output module;

[0053] A data processing module is used to obtain and preprocess the patient's multimodal medical data to obtain a multimodal dataset;

[0054] The feature extraction and fusion module is used to extract and fuse features based on the multimodal data set to obtain multimodal features as the patient's current status;

[0055] The training set acquisition module is used to build a decision network and input the patient's current status to obtain the current diagnosis and treatment plan. Based on the patient's current status and the current diagnosis and treatment plan as the data group, the above process is repeated to obtain multiple data groups to form the training set;

[0056] The network optimization module is used to optimize the decision network based on the training set and the core objective function to obtain the optimized decision network;

[0057] The optimal solution output module is used to obtain the status of the patient to be recommended and input it into the optimized decision network to obtain the optimal diagnosis and treatment plan.

[0058] Example 2

[0059] The embodiment of the present invention discloses a multimodal diagnosis and treatment plan recommendation system based on reinforcement learning, comprising: a data processing module, a feature extraction and fusion module, a training set acquisition module, a network optimization module and an optimal plan output module.

[0060] The data processing module is used to obtain the patient's multimodal medical data and perform preprocessing to obtain a multimodal dataset.

[0061] Preferably, Figure 2 As shown, the data processing module includes: a data acquisition unit, a first preprocessing unit, a second preprocessing unit and a labeling processing unit;

[0062] A data acquisition unit is used to acquire the patient's multimodal medical data, which includes: medical report text, diagnosis and treatment dialogue text and medical images;

[0063] A first preprocessing unit is used to perform denoising, resolution adjustment, grayscale standardization, and data enhancement in sequence on the medical image to obtain a preprocessed medical image;

[0064] A second preprocessing unit is used to perform text cleaning and data enhancement based on the medical report text and the diagnosis and treatment dialogue text respectively, to obtain a preprocessed medical report text and a preprocessed diagnosis and treatment dialogue text respectively;

[0065] The labeling processing unit is used to assign the same labels to the preprocessed medical images, preprocessed medical report texts and preprocessed diagnosis and treatment conversation texts, and then perform standardization processing on them respectively, so as to obtain the final medical images, final medical report texts and final diagnosis and treatment conversation texts which together constitute a multimodal dataset.

[0066] Preferably, in this embodiment, the multimodal medical data of the patient is obtained from public medical datasets such as MIMIC, MedTrinity-25M, MedMNIST, and M3D-Cap.

[0067] Preferably, the first preprocessing unit is used to perform denoising based on the medical image to reduce noise interference in the image, then perform resolution adjustment, and then perform grayscale normalization, that is, standardize the pixel values ​​to ensure that the grayscale distribution between different images is relatively consistent, and finally perform data enhancement, that is, rotate, flip and other operations on the image to finally obtain a preprocessed medical image.

[0068] Preferably, the second preprocessing unit is used to perform text cleaning based on the medical report text and the diagnosis and treatment dialogue text to remove invalid characters and noise information, and then perform data enhancement, that is, perform synonym replacement and text generation and other operations to enhance the diversity of the training corpus, and obtain preprocessed medical report text and preprocessed diagnosis and treatment dialogue text accordingly.

[0069] Preferably, the annotation processing unit is used to assign the same label to each data instance in the preprocessed medical image, preprocessed medical report text and preprocessed diagnosis and treatment conversation text to ensure that the label information can be correctly aligned during multimodal fusion, and then perform standardization processing on each of them to obtain the final medical image, final medical report text and final diagnosis and treatment conversation text, which together constitute a multimodal data set.

[0070] Preferably, the label includes disease type, lesion area and disease severity.

[0071] The feature extraction and fusion module is used to extract and fuse features based on the multimodal data set to obtain multimodal features as the patient's current status.

[0072] Preferably, the feature extraction and fusion module includes: a feature extraction unit and a feature fusion unit;

[0073] A feature extraction unit is used to extract features based on the final medical report text, the final diagnosis and treatment dialogue text, and the final medical image, respectively, to obtain medical numerical features, medical text features, and medical image features;

[0074] The feature fusion unit is used to perform weighted fusion based on medical numerical features, medical text features and medical image features to obtain multimodal features.

[0075] Preferably, Figure 3 As shown, the feature extraction unit includes a first feature extraction subunit;

[0076] A first feature extraction subunit is used to extract all types of numerical data based on the final medical report text, and encode the numerical data to obtain numerical information;

[0077] It is also used to perform word segmentation based on the final medical report text and embed word vectors on the segmented text to obtain the semantic information of each word. Based on the corresponding semantic information and numerical information, medical numerical features are obtained.

[0078] Preferably, taking the diagnosis of diabetic retinopathy as an example, the first feature extraction subunit extracts various numerical data from the final medical report text, such as blood pressure, insulin level, fasting blood glucose level and glycosylated hemoglobin value; the extracted numerical data is encoded by a numerical encoder or a pre-trained Bio_ClinicalBERT dedicated medical model to convert it into a vectorized representation to obtain numerical information N embedding :

[0079] N embedding =Number encoder (N t );

[0080] Among them, Number encoder Indicates the numerical encoder, N t Represents numeric data.

[0081] Preferably, the first feature extraction subunit performs word segmentation based on the final medical report text, dividing the text into manageable vocabulary units. For example, the word segmentation of "blood pressure value 120 / 80mmHg" is ['blood pressure value', '120 / 80mmHg']. The word vectors of these segmented texts are embedded to generate the semantic information V of each word. text , based on the corresponding semantic information V text and numerical information N embedding Combined, we get the medical numerical feature H embedding :

[0082] H embedding =Combine(V text ; N embedding );

[0083] Here, Combine represents an information combining operation.

[0084] Preferably, the feature extraction unit further includes a second feature extraction subunit;

[0085] The second feature extraction subunit is used to perform word segmentation processing and text cleaning in sequence based on the final diagnosis and treatment dialogue text to obtain cleaned word segmentation text, perform speaker annotation based on the cleaned word segmentation text to obtain annotated word segmentation text, and convert the annotated word segmentation text into a vector representation to obtain medical text features.

[0086] Preferably, the second feature extraction subunit performs word segmentation processing on the final diagnosis and treatment dialogue text based on a text encoder or a pre-trained BioBERT dedicated medical model. For example, the patient's self-reported symptoms include thirst, polyuria, fatigue, etc., and then performs text cleaning to remove irrelevant symbols and noise information to obtain a cleaned word segmentation text. Then, the text is annotated based on the cleaned word segmentation text, that is, the speaker of the dialogue is annotated as a patient or a doctor. For example, the word segmentation corresponding to the patient's self-reported symptoms is annotated as a patient to obtain an annotated word segmentation text. Finally, the annotated word segmentation text is converted into a vector representation based on the text encoder to obtain the medical text feature T embedding :

[0087] T embedding =Text encoder (T t );

[0088] Among them, Text encoder represents the text encoder, T t is the input labeled segmented text.

[0089] Preferably, the feature extraction unit further includes a third feature extraction subunit;

[0090] a third feature extraction subunit, configured to perform image denoising and normalization operations on the 2D medical image in the final medical image to obtain a processed 2D medical image, and perform feature extraction on the processed 2D medical image based on the 2D image encoder to obtain 2D medical image features;

[0091] It is also used to perform image denoising and normalization operations based on the 3D medical image in the final medical image to obtain a processed 3D medical image, and to perform feature extraction on the processed 3D medical image based on the 3D image encoder to obtain 3D medical image features. The 2D medical image features and the 3D medical image features together constitute the medical image features.

[0092] Preferably, the 2D image encoder based on the convolutional neural network processes the processed 2D medical image I t Perform feature extraction, such as fundus color photos (fundus image features are used to identify the severity of diabetic retinopathy), to obtain 2D medical image features I embedding :

[0093] I embedding =Image encoder (It );

[0094] Among them, Image encoder Represents a 2D image encoder.

[0095] Preferably, a 3D image encoder based on a 3D convolutional neural network is used to process the processed 3D medical image I t_3D Perform feature extraction, such as optical coherence tomography (OCT, used to detect complications), extract its features, and obtain 3D medical image features I 3Dembedding :

[0096] I 3Dembedding =Image 3Dencoder (I t_3D );

[0097] Among them, Image 3Dencoder Represents a 3D image encoder.

[0098] Preferably, the feature fusion unit includes: a weighted feature acquisition subunit and a feature fusion subunit;

[0099] The weighted feature acquisition subunit is used to perform weighted fusion based on medical numerical features, medical text features and medical image features, and obtain numerical weighted features, text weighted features and image weighted features respectively;

[0100] The feature fusion subunit is used to fuse numerical weighted features, text weighted features and image weighted features to obtain multimodal features.

[0101] Preferably, the weighted feature acquisition subunit is used to perform weighted fusion of medical numerical features, medical text features, and medical image features based on the attention mechanism of the Transformer architecture, and obtain numerical weighted features, text weighted features, and image weighted features respectively:

[0102] Z num =Attention(H embedding ,α1);

[0103] Z text =Attention(T embedding ,α2);

[0104] Z image =Attention(I embedding ,α3);

[0105] Z 3Dimage =Attention(I 3Dembedding ,α4);

[0106] Among them, Znum represents the numerical weighted feature, Z text represents the text weighted feature, Z image Represents the weighted features of 2D medical images, Z 3Dimage represents the weighted features of 3D medical images, α i Represents the weight of the corresponding modality, i = 1, 2, 3, 4, and Attention represents the attention mechanism, which can automatically adjust the weight of each feature according to different tasks and contexts.

[0107] Preferably, the feature fusion subunit is used to fuse the numerical weighted features, text weighted features and image weighted features to obtain the multimodal feature Z fused :

[0108] Z fused =Fusion model (Z num ,Z text ,Z image ,Z 3Dimage );

[0109] Among them, Fusion model Represents the feature fusion model.

[0110] Preferably, the multimodal feature Z fused The patient's current state s in the state space is used as the subsequent reinforcement learning training process t :s t ={Z fused}.

[0111] The training set acquisition module is used to build a decision network and input the patient's current status to obtain the current diagnosis and treatment plan. Based on the patient's current status and the current diagnosis and treatment plan as the data group, the above process is repeated to obtain multiple data groups to constitute the training set.

[0112] Preferably, the training set acquisition module includes: a data group acquisition unit and a loop execution unit;

[0113] a data group acquisition unit, configured to construct a decision network and input the patient's current state to obtain a current diagnosis and treatment plan, obtain a new state of the patient based on the current diagnosis and treatment plan, obtain a current reward value based on the new state of the patient and the patient's current state, and use the patient's current state, the current diagnosis and treatment plan, the new state of the patient, and the current reward value as a data group;

[0114] The loop execution unit is used to loop through the above process of obtaining data groups to obtain multiple data groups that together constitute a training set.

[0115] Preferably, the decision network is an existing general network, and the specific network structure is not limited.

[0116] Preferably, the current diagnosis and treatment plan includes disease classification and combination of therapeutic drugs. Specifically, the disease classification is the diagnosis and prediction of the disease type, and the combination of therapeutic drugs is the selection of corresponding therapeutic antibiotics, including the dosage and administration cycle. The current diagnosis and treatment plan a t for:

[0117] a t ={Diagnosis 1, Treatment 1; Diagnosis 2, Treatment 2; Diagnosis 3, Treatment 3};

[0118] For the diagnosis of diabetic retinopathy in this embodiment, the current diagnosis and treatment plan a t for:

[0119] a t ={If it is diabetes, follow a low-carb diet; if it is diabetes, adjust medication; if it is not diabetes, follow an exercise plan}.

[0120] Preferably, the patient's new state s is obtained after the simulation is executed based on the current diagnosis and treatment plan t+1 , based on the patient's new status s t+1 and the patient's current status t Compare and evaluate previous treatment optionsa t The diagnostic accuracy and treatment effectiveness of the current reward value are obtained based on the diagnostic accuracy and treatment effectiveness.

[0121] Preferably, the current reward value R t for:

[0122] R t =ω1·diagnosis_accuracy+ω2·treatment_effectiveness;

[0123] Among them, ω1 and ω2 represent the first weight factor and the second weight factor respectively, dingnosis_accuracy represents the diagnosis accuracy, and tretment_effectiveness represents the treatment effectiveness.

[0124] Preferably, diagnostic accuracy is used to calculate the proportion of correct diagnoses:

[0125]

[0126] Treatment effectiveness is expressed as:

[0127]

[0128] Where N represents the total number of patients, that is, the number of patients participating in the evaluation of the treatment plan, ΔC i Indicates that the condition of the i-th patient has improved after treatment.

[0129] Preferably, the reward value is used to evaluate the effectiveness of each diagnosis and treatment plan. A correct diagnosis or effective treatment plan will receive a higher reward, while an incorrect diagnosis or treatment plan will be punished.

[0130] Preferably, based on the patient's current status t and current treatment optionsa t , patient's new status t+1 and the current reward value R t As a data set Dt: Dt = {(s t ,a t ,R t ,s t+1 )}.

[0131] Preferably, the above process of obtaining the data set is executed cyclically, that is, the patient's new state s t+1 Input into the decision network to obtain a new diagnosis and treatment plan a t+1 , based on the new diagnosis and treatment plana t+1 Execute to get the patient's new status s t+2 , based on the patient's new status s t+1 and the patient's new status t+2 Get the new reward value R t+1 , based on the patient's new status s t+1 、New diagnosis and treatment plana t+1 , patient's new status t+2 and the new reward value R t+1 As a new data set; then based on the patient's new status s t+2 Input into the decision network, and so on, execute in a loop to obtain a preset number of data groups to form a training set.

[0132] The network optimization module is used to optimize the decision network based on the training set and the core objective function to obtain the optimized decision network.

[0133] Preferably, the core objective function is specifically:

[0134]

[0135] Among them, argmax means maximization, E[L PPO (θ) represents the L PPO Average of (θ), L PPO (θ) represents the objective function, Indicates the average of samples in all time steps t, s t Indicates the patient's current status, a t represents the current diagnosis and treatment plan, D represents the data group, r t (θ) represents the probability ratio of the new and old strategies, represents the advantage function, and clip(·) represents r t (θ) is limited to the interval [1-∈, 1+∈] to prevent the update amplitude from being too large. min means taking the minimum value of the two terms in the objective function, which ensures that even if the policy update is too large, it will not significantly change the objective function. ∈ represents a hyperparameter.

[0136] Preferably, the probability ratio of the new and old strategies is r t (θ) represents the current policy π θ With the old strategy In the same condition t The ratio of the probability of choosing the same action under state s, that is, the current diagnosis and treatment plan and the previous diagnosis and treatment plan in state s t The ratio of the probability of choosing the same treatment plan under different conditions is used to reflect the difference in the tendency of treatment plans, specifically:

[0137]

[0138] Among them, when r t When (θ)>1, it means that the current strategy is more inclined to this diagnosis and treatment plan.

[0139] Preferably, the advantage function Used to measure the state t Next take a t The degree of goodness of the strategy relative to the average strategy is estimated by the generalized advantage estimate (GAE), which is expressed as:

[0140]

[0141] Among them, γ represents the discount factor, which is used to balance short-term and long-term rewards, λ represents the hyperparameter, which is used to control the balance between bias and variance, and l represents the offset of the time step, that is, the increment of l steps from the current time step t, which is used to calculate the temporal difference error δ of multiple time steps in the future. t+l The impact on the current advantage estimate can be specifically explained as the longer the time step, the smaller the impact of the diagnosis and treatment action;

[0142] δ t+l =r t +γV(s t+1 )-V(s t ), V(s t ) means in state s t According to the current strategy π θ The expected cumulative reward that can be obtained by continuing to execute, V(s t+1 ) means in the next state s t+1 According to the current strategy π θ The expected cumulative reward that can be obtained by continuing to execute, r tIt is the immediate reward obtained at time step t. If δ>0, it means that the actual diagnosis and treatment is better than expected.

[0143] Preferably, diagnostic accuracy, treatment effectiveness, and cumulative rewards are used as key evaluation indicators to evaluate the decision network;

[0144] Cumulative reward is used to measure the total reward obtained by the decision network during the entire decision-making process, reflecting the quality of the decision network's overall decision. A higher value indicates a better strategy. It is expressed as:

[0145]

[0146] Where T represents the total number of time steps in the training process.

[0147] Preferably, according to the evaluation results of the decision network, the strategy optimization process is adjusted to further improve the diagnostic accuracy and the effectiveness of the treatment plan. If the diagnostic accuracy is low, the multimodal medical feature extraction and fusion module is optimized to improve the diagnostic accuracy; if the treatment effectiveness is low, the reward function is adjusted to make it closer to clinical practice and improve the quality of treatment recommendations; if the cumulative reward is unstable, the hyperparameters and learning rate in the training process are adjusted to ensure the stability of the PPO strategy optimization.

[0148] The optimal solution output module is used to obtain the status of the patient to be recommended and input it into the optimized decision network to obtain the optimal diagnosis and treatment plan.

[0149] Preferably, obtain the status s of the patient to be recommended t1 Input to the optimized decision network Get the best diagnosis and treatment plan: For example, disease diagnosis: diabetes; treatment plan: low-carb diet, increase the dosage of hypoglycemic drugs.

[0150] Example 3

[0151] Based on the same inventive concept, the present invention further provides a computer device, comprising a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus;

[0152] Memory for storing computer programs;

[0153] The processor is used to execute a program stored in the memory, and can perform a multimodal diagnosis and treatment plan recommendation method based on reinforcement learning.

[0154] The electronic device may include: a processor, a communications interface, a memory, and a communications bus, wherein the processor, the communications interface, and the memory communicate with each other via the communications bus. The processor may call logic instructions in the memory to execute a multimodal diagnosis and treatment plan recommendation method based on reinforcement learning:

[0155] Acquire and preprocess the patient's multimodal medical data to obtain a multimodal dataset;

[0156] Feature extraction and fusion are performed based on multimodal datasets to obtain multimodal features as the patient's current status;

[0157] Construct a decision network and input the patient's current status to obtain the current diagnosis and treatment plan. Based on the patient's current status and the current diagnosis and treatment plan as the data group, the above process is repeated to obtain multiple data groups to form a training set;

[0158] Optimize the decision network based on the training set and the core objective function to obtain the optimized decision network;

[0159] The status of the patient to be recommended is input into the optimized decision network to obtain the optimal diagnosis and treatment plan.

[0160] In addition, the logical instructions in the aforementioned memory can be implemented in the form of a software functional unit and, when sold or used as an independent product, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present invention.

[0161] The aforementioned storage media include: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk and other media that can store program code.

[0162] It can be seen from the above technical solutions that, compared with the prior art, the present invention discloses a multimodal diagnosis and treatment plan recommendation system based on reinforcement learning, which has the following beneficial effects:

[0163] 1. The present invention adopts special architectures such as numerical encoders (numerical reports) and text encoders (diagnosis and treatment texts) to extract features based on the characteristics of different modal data, accurately obtaining the clinical significance of numerical indicators such as blood pressure and blood sugar, as well as the semantics of text entities such as disease names and symptom descriptions, significantly enhancing the model's ability to understand medical expertise.

[0164] 2. The present invention integrates the numerical data of medical reports, medical texts of diagnosis and treatment dialogues, and multimodal information of 2D / 3D medical images, combines the encoder architecture dedicated to each modality to extract corresponding features, and uses the Transformer architecture for cross-modal feature fusion, fully exploring the correlation between the data of each modality, overcoming the data information limitations of a single modality, and improving the accuracy and comprehensiveness of disease diagnosis.

[0165] 3. The present invention adopts reinforcement learning algorithm training model to optimize diagnosis and treatment strategies. The system can generate personalized diagnosis and treatment plans according to the patient's real-time status (multimodal feature vector), and continuously adjust the strategy through reward function (diagnostic accuracy, treatment effect) to improve treatment effectiveness and clinical adaptability.

[0166] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method description.

[0167] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A multimodal diagnosis and treatment plan recommendation system based on reinforcement learning, characterized by: include: Data processing module, feature extraction and fusion module, training set acquisition module, network optimization module and optimal solution output module; The data processing module is used to obtain and preprocess the patient's multimodal medical data to obtain a multimodal data set; The feature extraction and fusion module is used to extract and fuse features based on the multimodal data set to obtain multimodal features as the patient's current state; The training set acquisition module is used to construct a decision network and input the patient's current state to obtain the current diagnosis and treatment plan. Based on the patient's current state and the current diagnosis and treatment plan as a data group, the above process is repeated to obtain multiple data groups to form a training set; The network optimization module is used to optimize the decision network based on the training set and the core objective function to obtain an optimized decision network; The optimal solution output module is used to obtain the status of the patient to be recommended and input it into the optimized decision network to obtain the optimal diagnosis and treatment solution.

2. A multimodal diagnosis and treatment plan recommendation system based on reinforcement learning according to claim 1, characterized in that: The data processing module includes: a data acquisition unit, a first preprocessing unit, a second preprocessing unit and a labeling processing unit; The data acquisition unit is configured to acquire the multimodal medical data of the patient, wherein the multimodal medical data includes: medical report text, diagnosis and treatment dialogue text, and medical images; The first preprocessing unit is configured to sequentially perform denoising, resolution adjustment, grayscale standardization, and data enhancement on the medical image to obtain a preprocessed medical image; The second preprocessing unit is used to perform text cleaning and data enhancement based on the medical report text and the diagnosis and treatment dialogue text respectively, to obtain a preprocessed medical report text and a preprocessed diagnosis and treatment dialogue text respectively; The labeling processing unit is used to assign the same labels to the preprocessed medical images, the preprocessed medical report texts and the preprocessed diagnosis and treatment conversation texts, and then perform standardization processing on them respectively, so as to obtain the final medical images, the final medical report texts and the final diagnosis and treatment conversation texts which together constitute the multimodal dataset.

3. A multimodal diagnosis and treatment plan recommendation system based on reinforcement learning according to claim 2, characterized in that: The feature extraction and fusion module includes: a feature extraction unit and a feature fusion unit; The feature extraction unit is used to extract features based on the final medical report text, the final diagnosis and treatment dialogue text and the final medical image, respectively, to obtain medical numerical features, medical text features and medical image features; The feature fusion unit is used to perform weighted fusion based on the medical numerical feature, the medical text feature and the medical image feature to obtain the multimodal feature.

4. A multimodal diagnosis and treatment plan recommendation system based on reinforcement learning according to claim 3, characterized in that: The feature extraction unit includes a first feature extraction subunit; The first feature extraction subunit is configured to extract all types of numerical data based on the final medical report text, and perform encoding based on the numerical data to obtain numerical information; It is also used to perform word segmentation based on the final medical report text and embed word vectors on the segmented text to obtain semantic information of each word, and obtain the medical numerical features based on the corresponding combination of the semantic information and the numerical information.

5. The multimodal diagnosis and treatment plan recommendation system based on reinforcement learning according to claim 3 is characterized in that: The feature extraction unit further includes a second feature extraction subunit; The second feature extraction subunit is used to perform word segmentation processing and text cleaning in sequence based on the final diagnosis and treatment dialogue text to obtain a cleaned word segmentation text, perform speaker annotation based on the cleaned word segmentation text to obtain annotated word segmentation text, and convert the annotated word segmentation text into a vector representation to obtain the medical text feature.

6. The multimodal diagnosis and treatment plan recommendation system based on reinforcement learning according to claim 3 is characterized in that: The feature extraction unit further includes a third feature extraction subunit; The third feature extraction subunit is configured to perform image denoising and normalization operations on the 2D medical image in the final medical image to obtain a processed 2D medical image, and perform feature extraction on the processed 2D medical image based on a 2D image encoder to obtain 2D medical image features; It is also used to perform image denoising and normalization operations based on the 3D medical image in the final medical image to obtain a processed 3D medical image, and to perform feature extraction on the processed 3D medical image based on a 3D image encoder to obtain 3D medical image features. The 2D medical image features and the 3D medical image features together constitute the medical image features.

7. The multimodal diagnosis and treatment plan recommendation system based on reinforcement learning according to claim 3 is characterized in that: The feature fusion unit includes: a weighted feature acquisition subunit and a feature fusion subunit; The weighted feature acquisition subunit is used to perform weighted fusion based on the medical numerical features, the medical text features and the medical image features, respectively, to obtain numerical weighted features, text weighted features and image weighted features respectively; The feature fusion subunit is used to fuse the numerical weighted features, the text weighted features and the image weighted features to obtain the multimodal features.

8. The multimodal diagnosis and treatment plan recommendation system based on reinforcement learning according to claim 1, characterized in that: The training set acquisition module includes: a data group acquisition unit and a loop execution unit; The data group acquisition unit is used to construct a decision network and input the patient's current state to obtain a current diagnosis and treatment plan, obtain a new patient state based on the current diagnosis and treatment plan, obtain a current reward value based on the new patient state and the patient's current state, and use the patient's current state, the current diagnosis and treatment plan, the new patient state, and the current reward value as a data group; The loop execution unit is used to loop through the process of obtaining the data group, so as to obtain a plurality of data groups that together constitute the training set.

9. The multimodal diagnosis and treatment plan recommendation system based on reinforcement learning according to claim 8, characterized in that: The current reward value R t for: R t =ω1·diagnosis_accuracy+ω2·treatment_effectiveness; Among them, ω1 and ω2 represent the first weight factor and the second weight factor respectively, diagnosis_accuracy represents the diagnosis accuracy, and treatment_effectiveness represents the treatment effectiveness.

10. The multimodal diagnosis and treatment plan recommendation system based on reinforcement learning according to claim 1, characterized in that: The core objective function is specifically: Among them, argmax means maximization, E[L PPO (θ)] represents the L PPO Average of (θ), L PPO (θ) represents the objective function, It means taking the average of samples in all time steps t, r t (θ) represents the probability ratio of the new and old strategies, represents the advantage function, and clip(·) represents r t (θ) is restricted to the interval [1-∈, 1+∈], where ∈ represents a hyperparameter.

Images