Method, device and medium for improving the safety of auxiliary medical decision-making system

Through the inverse constraint reinforcement learning model that blocks self-attention by causally, a cost function that can capture long historical state sequences is constructed, which solves the limitations of cost function design in the medical field in the existing technology, and realizes safe medical care in medical scenarios Improve security of strategy learning and decision-making systems.

CN118629636BActive Publication Date: 2024-11-26UNIV OF SCI & TECH OF CHINA
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411104622.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-13
Publication Date
2024-11-26
Estimated Expiration
2044-08-13

AI Technical Summary

Technical Problem

The existing cost function design has limitations in the medical field and cannot be effectively applied to auxiliary medical decision-making systems, resulting in unsafe drug dosage and parameter settings, and the inverse constraint reinforcement learning method cannot be directly used in the medical environment.

Method used

Using an inverse constraint reinforcement learning model based on causal blocking of self-attention, by obtaining medical expert data sets and generating violation data sets, a cost function that can capture long historical state sequences is constructed, and a strategy used to optimize medical decision-making systems is avoided directly with patients. Environmental interaction.

Benefits of technology

It realizes safe medical strategy learning in medical scenarios, improves the safety of auxiliary medical decision-making systems, avoids unsafe drug dosage and parameter settings, enhances the interpretability of the model, and is suitable for offline applications in the medical field.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118629636B_ABST
    Figure CN118629636B_ABST
Patent Text Reader

Abstract

The present invention discloses a method, device and medium for improving the safety of an auxiliary medical decision-making system. The method comprises: step 1, constructing an inverse constraint reinforcement learning model based on a self-attention model: adopting a causal shielding self-attention model, taking medical expert data and violation data with a long historical state sequence of patients as input, and outputting the patient's action penalty, and obtaining a cost function gradient equation that minimizes the medical expert data constraint and maximizes the violation data constraint; step 2, learning a constraint strategy with a cost function: training the inverse constraint reinforcement learning model based on causal shielding self-attention in step 1 to obtain a cost function that can observe a long historical sequence and capture key states, and using the cost function to optimize the strategy constraint with an existing constraint reinforcement learning algorithm of a medical decision-making system, and obtaining a safe medical decision-making system. The method realizes the learning of the safety strategy of the auxiliary medical decision-making system in a medical scenario, and improves the safety of the auxiliary medical decision-making system.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of auxiliary medical decision-making systems that utilize artificial intelligence to assist in decision-making, and in particular to a method, device and medium for improving the security of auxiliary medical decision-making systems. Background Art

[0002] In recent years, AI-assisted treatment has gained increasing attention as a solution. AI assistants monitor the patient's condition and directly provide nurses with recommendations on drug dosage or device parameters. Compared with traditional treatments, AI assistants provide timely diagnosis, personalized treatment, and reduce dependence on experienced doctors. Therefore, it is crucial to develop an effective medical assistance AI assistant.

[0003] However, the current application of reinforcement learning (RL) in healthcare still has some unsafe behaviors, including excessive drug doses, medical parameter settings, or sudden changes in drug doses. These "excessive" or "mutated" behaviors can cause significant pain to patients, leading to acute hypotension, hypertension, or arrhythmias, resulting in organ damage and potentially fatal consequences. Taking sepsis treatment strategy learning as an example, some studies have shown that the mortality rate of patients receiving vasopressors at doses exceeding 1 μg / kg / min is as high as 90%. In addition, mutated vasopressors can have a rapid effect on blood vessels, leading to acute changes in blood pressure, which threatens the patient's life. The application of RL in sepsis does not consider these safety issues. Komorowski and Raghu et al. proposed using RL to dynamically recommend intravenous (IV) and vasopressors (Vaso) for sepsis patients to achieve sepsis strategy learning in offline scenarios. Huang et al. used an RL method based on the deep decision policy gradient (DDPG) algorithm to solve the limitations of discrete actions in treatment recommendations. However, even these advanced DDPG methods still have a high probability of "excessive" and "mutated" unsafe behaviors in Vaso recommendations.

[0004] People hope that the auxiliary medical decision-making system can achieve safe medical strategy learning to avoid unsafe behaviors, such as excessive or sudden drug doses. However, some current constrained reinforcement learning methods adopt a constrained approach, that is, while maximizing rewards, actions that violate constraints are limited to a certain range. The design of the cost function in the application of this method is crucial and determines the effectiveness of the constrained reinforcement learning method. Regarding the design of the cost function, a direct method is to use medical prior knowledge to represent the cost function. For example, in the treatment of sepsis, fluids greater than 2000ml / 4h or vasopressors greater than 1µg / kg / min are usually considered unsafe behaviors to define the cost function.

[0005] However, this custom constraint may lead to excessive suppression of drug dosage. This is mainly because the custom cost function has the following problems: (1) the constraint strength is the same for excessive drug dosage; (2) it lacks personalization, and each patient has different tolerance to the maximum dose, so it cannot be generalized. In addition, the custom cost function lacks generalization. For example, it needs to be redefined for other medical or safety issues, and the requirement for prior knowledge is high, which limits its application in safe medical strategy learning.

[0006] Another way to describe the cost function is inverse constraint reinforcement learning. This method can infer the constraints that experts follow from their demonstrations. Inverse constraint reinforcement learning uses a network to fit the cost function and does this by alternating between updating the policy and the cost function. However, the inverse constraint reinforcement learning method cannot be directly applied to the medical field at present, mainly due to the following challenges:

[0007] (1) Application of Markov properties in medical settings is limited:

[0008] The inverse constraint reinforcement learning method relies on the Markov property, that is, the next state of the system is only related to the current state, and has nothing to do with the historical state. This requires that the current state should contain all historical information. However, in the medical field, the state we define only contains the state of the patient at a specific moment, and does not contain historical information. Therefore, using this state definition method, the inverse constraint reinforcement learning method cannot obtain the patient's long-term historical information, thereby ignoring the individual differences of patients, resulting in limited effectiveness;

[0009] (2) Modeling complex medical interactions online is not feasible:

[0010] On the other hand, the core goal of the inverse constraint reinforcement learning method is to minimize the constraint value on the medical expert data, while maximizing the cost function value on the violation data set. The violation data set is obtained through continuous interaction between the strategy and the environment to understand the reaction of the environment under the current strategy. However, in the medical environment, the environment is the patient itself. For safety reasons, the strategy cannot be directly applied to the patient to obtain real feedback. Therefore, the inverse constraint reinforcement learning method cannot be applied to the medical environment.

[0011] Therefore, existing cost function design methods have certain limitations in the medical field and cannot be effectively applied to medical scenarios to realize security strategy learning of auxiliary medical decision-making systems.

[0012] In view of this, the present invention is proposed. Summary of the invention

[0013] The purpose of the present invention is to provide a method, device and medium for improving the security of an auxiliary medical decision-making system, which can realize security strategy learning of the auxiliary medical decision-making system in a medical scenario, improve the security of the auxiliary medical decision-making system, and thus solve the above-mentioned technical problems existing in the prior art.

[0014] The objective of the present invention is achieved through the following technical solutions:

[0015] A method for improving the safety of a medical decision-making assistance system, comprising:

[0016] Step 1, obtaining a medical expert dataset and generating a violation dataset through a self-attention-based generative model, wherein both the medical expert dataset and the violation dataset contain a long historical state sequence of each patient;

[0017] Step 2: Build an inverse constraint reinforcement learning model based on causal shielding self-attention:

[0018] The causal shielding self-attention model is adopted, with the long history state sequence of a patient in the medical expert data set obtained in step 1 and the long history state sequence of a patient in the violation data set generated in step 1 as input, and the action penalty of the patient as output, to obtain the optimization objective equation of the inverse constraint reinforcement learning model based on causal shielding self-attention. Minimizing the objective equation is to obtain the gradient equation of the cost function that minimizes the constraints on the medical expert data and maximizes the constraints on the violation data, which is:

[0019] ;

[0020] in, represents the optimization parameters of the inverse constrained reinforcement learning model based on causal masked self-attention; Indicates is the gradient operator of the parameter; represents the cost function calculation equation; Indicates taking the average value; represents a long historical state sequence of a patient from the violation dataset; represents the medical experts dataset; represents the logarithmic function; represents the cost function for violating the dataset, i.e., the penalty for the patient’s actions; Represents a long historical state sequence of a patient from a medical expert dataset; Indicates violation of the data set; Represent the cost function of the medical expert data;

[0021] Step 3: Apply the cost function obtained in step 2 to learn the constraint strategy:

[0022] Through step 2, an inverse constrained reinforcement learning model based on causal masked self-attention is obtained. The inverse constrained reinforcement learning model based on causal masked self-attention is trained to obtain a cost function that can observe the patient's long historical state sequence and capture key states. The cost function is used in the existing constrained reinforcement learning algorithm of the medical decision-making system to constrain optimization of the strategy to obtain a safe medical decision-making system.

[0023] A processing device, comprising:

[0024] at least one memory for storing one or more programs;

[0025] At least one processor can execute one or more programs stored in the memory. When the one or more programs are executed by the processor, the processor can implement the method described in the present invention.

[0026] A readable storage medium stores a computer program, which can implement the method described in the present invention when the computer program is executed by a processor.

[0027] Compared with the prior art, the method, device and medium for improving the safety of the auxiliary medical decision-making system provided by the present invention have the following beneficial effects:

[0028] By using the long sequence observation mechanism based on causal shielding self-attention to model the cost function, a doctor-like observation sequence judgment mechanism in medical scenarios is realized, which can capture key state information and increase the interpretability of the model. We consider offline scenarios, use a self-attention-based generative model to generate violation data, and directly extract medical expert data from the medical database. Using the above two datasets, we analyze the essence of inverse constraint reinforcement learning and design a gradient function to train the cost function. Compared with the previous simple network of inverse constraint reinforcement learning, an offline inverse constraint reinforcement learning algorithm is realized for the first time. This method is more suitable for use in medical scenarios and avoids direct interaction with the patient environment, ensuring the feasibility of the method. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0030] Figure 1 A flow chart of a method for improving the security of a medical assistance decision-making system provided in an embodiment of the present invention.

[0031] Figure 2A schematic diagram of the composition of an inversely constrained reinforcement learning model based on causal masked self-attention according to the method provided in an embodiment of the present invention.

[0032] Figure 3 A schematic diagram of the composition of a self-attention-based generation model of the method provided in an embodiment of the present invention.

[0033] Figure 4 A schematic diagram of a process for generating a violation data set according to a method provided in an embodiment of the present invention.

[0034] Figure 5 A relationship diagram between the SOFA score and the cost function in the effectiveness performance of capturing unsafe states in a sepsis environment is provided for the method provided by the embodiment of the present invention.

[0035] Figure 6 The method provided by the embodiment of the present invention is capable of capturing the relationship between the lactate level and the cost function in the effectiveness performance of the unsafe state in the sepsis environment.

[0036] Figure 7 A relationship diagram between mean arterial pressure and cost function in the effectiveness performance of capturing unsafe states in a sepsis environment is provided for the method provided by an embodiment of the present invention.

[0037] Figure 8 The method provided by the embodiment of the present invention is capable of capturing the relationship between the heart rate and the cost function in the effectiveness performance of the unsafe state in a sepsis environment.

[0038] Fig. 9 A schematic diagram of the relationship between the penalty value and the mortality rate in the method provided in an embodiment of the present invention.

[0039] Fig.10 A graph showing the relationship between the difference in vasopressor dosage and mortality between the model strategy and the doctor strategy in the comparison of the effects of the method provided in an embodiment of the present invention and the existing DDPG algorithm.

[0040] Fig.11 A graph showing the relationship between the difference in intravenous infusion dosage and mortality rate between the model strategy and the doctor strategy in the effect comparison between the method provided in an embodiment of the present invention and the existing DDPG algorithm. DETAILED DESCRIPTION

[0041] The following is a clear and complete description of the technical solutions in the embodiments of the present invention in combination with the specific content of the present invention; it is obvious that the described embodiments are only part of the embodiments of the present invention, not all of the embodiments, which does not constitute a limitation of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the protection scope of the present invention.

[0042] First, the terms that may be used in this article are explained as follows:

[0043] The term “and / or” means that either or both of them can be realized at the same time. For example, X and / or Y means both “X” or “Y” and “X and Y”.

[0044] The terms "include", "comprises", "contains", "has" or other descriptions with similar semantics should be interpreted as non-exclusive inclusion. For example, including certain technical feature elements (such as raw materials, components, ingredients, carriers, dosage forms, materials, dimensions, parts, components, mechanisms, devices, steps, procedures, methods, reaction conditions, processing conditions, parameters, algorithms, signals, data, products or products, etc.) should be interpreted as including not only certain technical feature elements explicitly listed, but also other technical feature elements known in the art that are not explicitly listed.

[0045] The term "consisting of..." means excluding any technical feature elements not explicitly listed. If this term is used in a claim, it will make the claim closed, so that it does not contain technical feature elements other than the technical feature elements explicitly listed, except for the conventional impurities related to them. If this term only appears in a clause of a claim, it only limits the elements explicitly listed in the clause, and the elements recorded in other clauses are not excluded from the overall claim.

[0046] Unless otherwise specified or limited, the terms "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense, for example: it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be an indirect connection through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in this article can be understood according to specific circumstances.

[0047] The orientation or position relationship indicated by terms such as "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise", etc. are based on the orientation or position relationship shown in the drawings and are only for the convenience and simplification of description, and do not explicitly or implicitly indicate that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation of this document.

[0048] The scheme provided by the present invention is described in detail below. The contents not described in detail in the embodiments of the present invention belong to the prior art known to professionals in the field. If the specific conditions are not specified in the embodiments of the present invention, they are carried out according to the conventional conditions in the field or the conditions recommended by the manufacturer. If the manufacturers are not specified in the reagents or instruments used in the embodiments of the present invention, they are all conventional products that can be purchased commercially.

[0049] like Figure 1 As shown, an embodiment of the present invention provides a method for improving the safety of a medical decision-making assistance system, comprising:

[0050] Step 1, obtaining a medical expert dataset and generating a violation dataset through a self-attention-based generative model, wherein both the medical expert dataset and the violation dataset contain a long historical state sequence of each patient;

[0051] Step 2: Build an inverse constraint reinforcement learning model based on causal shielding self-attention:

[0052] The causal shielding self-attention model is adopted, with the long history state sequence of a patient in the medical expert data set obtained in step 1 and the long history state sequence of a patient in the violation data set generated in step 1 as input, and the action penalty of the patient as output, to obtain the optimization objective equation of the inverse constraint reinforcement learning model based on causal shielding self-attention. Minimizing the objective equation is to obtain the gradient equation of the cost function that minimizes the constraints on the medical expert data and maximizes the constraints on the violation data, which is:

[0053] ;

[0054] in, represents the optimization parameters of the inverse constrained reinforcement learning model based on causal masked self-attention; Indicates is the gradient operator of the parameter; represents the cost function calculation equation; Indicates taking the average value; represents a long historical state sequence of a patient from the violation dataset; represents the medical experts dataset; represents the logarithmic function; represents the cost function for violating the dataset, i.e., the penalty for the patient’s actions; Represents a long historical state sequence of a patient from a medical expert dataset; Indicates violation of the data set; Represent the cost function of the medical expert data;

[0055] Step 3: Apply the cost function obtained in step 2 to learn the constraint strategy:

[0056] Through step 2, an inverse constrained reinforcement learning model based on causal masked self-attention is obtained. The inverse constrained reinforcement learning model based on causal masked self-attention is trained to obtain a cost function that can observe the patient's long historical state sequence and capture key states. The cost function is used in the existing constrained reinforcement learning algorithm of the medical decision-making system to constrain optimization of the strategy to obtain a safe medical decision-making system.

[0057] Preferably, in step 1 of the above method, the medical expert dataset is obtained by using data from an existing medical dataset or data directly obtained from a hospital.

[0058] Preferably, in step 1 of the above method, the medical expert dataset is obtained by using data from an existing medical dataset or data directly obtained from a hospital in the following manner, including:

[0059] Using data from existing medical datasets or data directly obtained from hospitals, the medical expert dataset is composed of long historical state sequences of multiple patients collected in the following manner:

[0060] exist time step, the state of a patient is represented as , the drug dosage taken by the hospital doctor for the patient's condition is expressed as And the next state of the patient after receiving the dose of the drug is expressed as and the reward is expressed as ; Repeat this process to obtain the treatment trajectory of a patient That is, the patient's long history state sequence, which is composed of multiple long history state sequences of multiple patients collected by records. , ,in, Represents treatment trajectory Serial number, Represents the number of patients in the medical specialists dataset.

[0061] Preferably, in step 1 of the above method, the generation model based on self-attention includes:

[0062] Input linear layer, causal attention layer, action prediction output linear layer, reward prediction output linear layer and next state prediction output linear layer; among them,

[0063] The input linear layer is communicatively connected to the input end of the causal self-attention layer, and can take the long historical state sequence of the patient from the medical expert data set as input, and encode the input long historical state sequence to obtain the historical state encoding;

[0064] The causal attention layer is respectively connected to the action prediction output linear layer, the reward prediction output linear layer and the next state prediction output linear layer, and can receive the historical state code output by the input linear layer, compress and learn the historical state code, and obtain the historical state feature;

[0065] The action prediction output linear layer can receive the historical state features output by the causal attention layer and predict the action based on the historical state features;

[0066] The reward prediction output linear layer is communicatively connected to the output end of the action prediction output linear layer, can receive the historical state features output by the causal attention layer and the action prediction output by the action prediction output linear layer, and predict the reward based on the historical state features and the action prediction;

[0067] The next state prediction output linear layer is communicatively connected to the output end of the action prediction output linear layer, can receive the historical state features output by the causal attention layer and the action prediction output by the action prediction output linear layer, and predict the next state based on the historical state features and the action prediction.

[0068] Preferably, in the above method, the action prediction output linear layer predicts the action according to the historical state characteristics in the following manner, including:

[0069] The action prediction output linear layer By minimizing the negative log-likelihood loss as the optimization objective, while maximizing the weight Entropy to predict actions ,for:

[0070] ;

[0071] in, Indicates taking the minimum value; The action prediction output linear layer representing the predicted action Parameters; represents the average value; Before Time step reward, state and previous The actions at each time step are represented as , , ; represents the medical experts dataset; represents the logarithmic function; represents the output of action prediction; represents the weight parameter; represents the Shannon entropy regularizer to avoid overfitting of action outputs;

[0072] The reward prediction output linear layer Perform reward prediction and the next state prediction output linear layer The next state prediction is formalized according to the following objectives:

[0073] ;

[0074] in, Indicates taking the minimum value; and They represent the next state prediction output linear layer parameters for predicting the next state and the reward prediction output linear layer parameters for predicting the reward respectively; represents the average value; Indicates The state of the time step; Indicates The reward for the time step; express and the action prediction output layer in the Predicted action at time step The union of; represents the medical experts dataset; and They respectively represent the next state prediction output linear layer that predicts the next state and the reward prediction output linear layer that predicts the current reward.

[0075] Preferably, in step 1 of the above method, the violation data set is generated based on the self-attention generation model in the following manner, including:

[0076] Step 121, taking the maximum reward of the long history sequence of patients in the medical expert dataset as the initial target reward and the initial state As input to the self-attention based generative model, the self-attention based generative model predicts the action in an autoregressive manner ;

[0077] Step 122: Initial target reward , initial state and predicted actions As input to the self-attention based generative model, the predicted reward and the next state ;

[0078] Step 123, gradually reduce the target reward ,Right now ,in, Indicates Target reward for repetitions; Indicates +1 target bonus for repetition; Indicates Repeated prediction reward;

[0079] Step 124, gradually derive the subsequent treatment trajectory changes to obtain the treatment trajectory ,in, is the initial state; Actions that are predicted; Rewards for predictions; is the predicted next state;

[0080] Step 125, repeat steps 121 to 124, with different Initial state As the input of the self-attention based generative model, we can get Different treatment trajectories , useful Different treatment trajectories Composition violation data set , , Represents treatment trajectory Serial number, Represents the number of patients in the violation dataset.

[0081] Preferably, in step 2 of the above method, the causal shielding self-attention model is composed of a causal self-attention layer and an attention layer connected in sequence; wherein,

[0082] The causal self-attention layer can take the patient's long historical state sequence as input. All information before the time step is compressed into historical state features ;

[0083] The attention layer is communicatively connected to the output end of the causal self-attention layer, and can output the historical state features of the causal self-attention layer. The cost function is the weighted sum of input and output.

[0084] Preferably, in step 2 of the above method, the causal self-attention layer uses a linear layer to use matrix multiplication to convert each historical state feature Project to different subspaces to generate query values , key value and cost value ,in, represents the time step; Represents the total length of the patient's long history state sequence;

[0085] The causal self-attention layer converts the query value into and key value Obtain weight representation through matrix calculation ,in, express The weight of the time step, express The cost of the time step; Represents the total length of the patient's long history state sequence; represents the query value, which is the causal self-attention layer in the The information that the time step hopes to obtain from other locations; Indicates the key value, which is The information characteristics actually contained in the time step;

[0086] The cost function is obtained by taking the weighted sum of the costs at each time step and expressing the final cost at each time step. , , where the meaning of each parameter is: represents the sum function; represents the time step; The total length of the sequence representing the patient's long history state sequence; express The weight of the time step; express The cost of a time step.

[0087] The embodiment of the present invention further provides a processing device, including:

[0088] at least one memory for storing one or more programs;

[0089] At least one processor can execute one or more programs stored in the memory, and when the one or more programs are executed by the processor, the processor can implement the above method.

[0090] The embodiments of the present invention further provide a readable storage medium storing a computer program, which can implement the above method when the computer program is executed by a processor.

[0091] In summary, the method of the embodiment of the present invention can be used as a constrained reinforcement learning algorithm that matches the medical scenario, thereby improving the effectiveness of the auxiliary medical decision-making system strategy and the security of the strategy, and enhancing the safe application of reinforcement learning technology in the medical field.

[0092] In order to more clearly demonstrate the technical solution and technical effects provided by the present invention, the solution provided by the embodiment of the present invention is described in detail with specific embodiments below.

[0093] Example 1

[0094] This embodiment provides a method for improving the safety of the auxiliary medical decision-making system, which is an offline inverse constraint reinforcement learning algorithm based on causal shielding self-attention, which can be applied to constraint reinforcement learning matching medical scenarios, thereby improving the effectiveness and safety of the auxiliary medical decision-making system strategy, and improving the safe application of reinforcement learning technology in the medical field. Figure 1 The components of learning a constrained strategy, the design of the cost function and the application of constrained reinforcement learning are demonstrated. The design of the cost function includes three parts, namely, inverse constraint reinforcement learning modeling based on causal masked self-attention, generation of violation data set and acquisition of medical expert data set. Constrained reinforcement learning is trained using the cost function obtained in the previous step to obtain a safe strategy.

[0095] First of all, the present invention is divided into three parts according to needs, including the model construction of inverse constraint reinforcement learning based on causal shielding self-attention, the acquisition of medical expert data set and the generation of violation data set. Among them, the former is to solve the challenge of limited application of Markov properties in medical environments, observe long sequence information based on the architecture of causal shielding self-attention, and imitate the observation of human doctors. The latter two are to solve the challenge that online modeling of complex medical interactions is not feasible, so that the model is trained on an offline data set to obtain the cost function of the medical environment. Finally, the learned cost function is applied to the currently advanced constrained reinforcement learning algorithm to realize the learning of safety strategies for auxiliary medical decision-making systems.

[0096] Step 1: Construction of an inverse constraint reinforcement learning model based on causal shielding self-attention:

[0097] The transformer is introduced into inverse constrained reinforcement learning to solve the observation limitation problem of the previous Markov property. Specifically, the transformer architecture with causal shielding self-attention is adopted, and the given length is The trajectory segment , co-embedded Inputs, that is, each position contains two embeddings of state and action, which are learned by linear layers and normalization. Then, the embeddings are input into the inverse constraint reinforcement learning model based on causal masked self-attention, and the output historical state features are generated. Among them, the historical state characteristics Value from the previous The historical state features are embedded into the decision Although the representation as a penalty function can provide historical information, it does not clearly explain which previous critical (or fatal) behavior caused the increase in penalty dt. In the medical field, the explanation of such critical behavior is crucial because it helps to enhance the interpretability of the model. To explain this, the present invention introduces an additional attention layer. This attention layer is used to define the penalty weights and of the non-Markov structure. It receives the output embedding of the causal transformer as input and generates the corresponding penalty and importance weights The output of the attention layer is obtained by dividing the value by the query With key The normalized dot product between them is weighted:

[0098] ;

[0099] Here, the softmax function is a commonly used activation function. Represents the sum function, keyword , Query Sum ,in, is the embedding dimension, and represents the time step; T is the length of the trajectory. In addition, for each time step , due to the historical state characteristics It depends only on the previous state-action pair and is used as the input embedding of the attention layer, so the penalty Only with time step The cost function is related to the previous time step. It is expressed as a weighted sum as follows:

[0100] ;

[0101] It is also possible to determine the time step The cost function value of each previous subsequence at the beginning; finally, the newly defined cost function is introduced and the gradient equation of the cost function is redefined as:

[0102] ;

[0103] in, represents the optimization parameters of the inverse constrained reinforcement learning model based on causal masked self-attention; Indicates is the gradient operator of the parameter; represents the cost function calculation equation; Indicates taking the average value; represents the logarithmic function; represents the medical experts dataset; Represents a long historical state sequence of a patient from a medical expert dataset; Indicates violation of the data set; represents a long historical state sequence of a patient from the violation dataset; Represent the cost function of the medical expert data; represents the cost function for violating the dataset, i.e., the penalty for the patient’s actions.

[0104] This statement means that the design goal is to minimize the constraints on the medical expert data and maximize the constraints on the violation data. In order to evaluate the equation offline, the present invention constructs an expert dataset and a violation dataset, respectively, using the following methods (2) and (3).

[0105] The above-mentioned inverse constraint reinforcement learning model based on causal shielding self-attention is composed as follows Figure 2 As shown, it includes a causal self-attention layer and an attention layer.

[0106] (2) Acquisition of medical expert datasets:

[0107] Medical expert datasets can be obtained by using existing medical datasets (e.g., the Multi-parameter Intelligent Monitoring in Intensive Care III (MIMIC-III) database) or directly from hospitals. Time step, the patient's state is recorded as , the drug dosage measures taken by hospital doctors according to the patient's condition , and the patient's next state after receiving this dose of the drug and rewards This process is repeated to obtain the patient's trajectory. As a long historical state sequence of patients, by observing multiple patients, multiple trajectories can be collected to form a medical expert dataset ,in, Represents treatment trajectory Serial number, Represents the number of patients in the medical specialists dataset.

[0108] (3) Generative model based on self-attention:

[0109] like Figure 3 As shown, Figure 3 The modeling idea of ​​generating a generative model for a violation dataset is shown. A self-attention-based generative model structure is used to generate states, rewards, and actions. Expert data is used as input. The goal is to minimize the output gap. The violation dataset can be expressed as , and the key to generating the model is to simulate the patient in the state Take action The next state after A simple method was designed by using the treatment trajectories in the medical expert dataset As input, and the self-attention architecture is used for data generation. Since Liu et al. have proved that the decision transformer does produce behaviors that violate constraints, the present invention designs a generative model based on self-attention. The optimization objective aims to minimize the negative log-likelihood loss of the action:

[0110] ;

[0111] in, Indicates taking the minimum value; is the action prediction output linear layer that predicts the action Parameters; Indicates taking the average value; It is before Time step reward, state and previous The actions at each time step are represented as , , ; It is a medical expert dataset; represents the logarithmic function; is the output of action prediction; represents the weight parameter; is the Shannon entropy regularizer to avoid overfitting of action outputs. By introducing the Shannon entropy regularizer To prevent overfitting and enhance robustness;

[0112] At the same time, the optimization objective needs to minimize the mean square error between the reward and the next state, which is:

[0113] ;

[0114] in, Indicates taking the minimum value; represents the average value; and They represent the next state prediction output linear layer parameters for predicting the next state and the reward prediction output linear layer parameters for predicting the reward respectively; and They represent the next state prediction output linear layer that predicts the next state and the reward prediction output linear layer that predicts the current reward respectively; Indicates The state of the time step; Indicates The reward for the time step, express and the action prediction output layer in the Predicted action at time step The union of; Represents the medical experts dataset.

[0115] Regarding the generation of violation data Figure 4 As shown, using the above generation model, take the target reward and the initial state As input, the treatment trajectory is generated autonomously in an autoregressive manner To prevent excessive error accumulation, generate data with a length of no more than 10; repeat this process times, we will get the violation data to form the violation data set , Represents treatment trajectory Serial number; Represents the number of patients in the violation dataset.

[0116] Based on the above steps, we can get an inverse constraint reinforcement learning algorithm based on causal shielding self-attention. By training this algorithm, we can get a cost function that can observe long historical sequences and capture key states. Then, this cost function is used in the existing constraint reinforcement learning algorithm of the medical decision-making system to achieve constraint optimization of the countermeasure path and realize safety strategy learning in the medical field.

[0117] Aiming at the problem that the existing inverse constraint reinforcement learning cannot be applied in the medical field, the present invention proposes a new inverse constraint reinforcement learning scheme to design a reasonable cost function and apply the constrained reinforcement learning algorithm, so as to optimize medical decision-making and effectively avoid unsafe strategies.

[0118] First, the method of the present invention utilizes a transformer-based long sequence observation mechanism and an attention mechanism to model the cost function, realizing an observation sequence judgment mechanism similar to that of a doctor in a medical scenario, and is able to capture key state information, thereby increasing the interpretability of the model. Considering offline scenarios, a generative model is used to generate radical data, and expert data is directly extracted from the medical database. Using the above two data sets, the essence of inverse constrained reinforcement learning is analyzed, and a gradient function is designed to train the cost function. Compared with the previous simple network of inverse constrained reinforcement learning, an offline quasi-constrained reinforcement learning algorithm is implemented for the first time. This method is more suitable for use in medical scenarios, and avoids direct interaction with the patient environment, ensuring the feasibility of the method.

[0119] Secondly, the method of the present invention is introduced into the constrained reinforcement learning algorithm. The combination of the two methods realizes an effective constraint strategy and optimizes the medical strategy, making the medical strategy safer and avoiding the unsafe behaviors of "too high" or "mutation" in the previous methods. Moreover, the process of the present invention further proves that the transformer-based architecture method has better effects in medical scenarios.

[0120] Example 2

[0121] See also Figure 1 This embodiment provides a method for improving the security of an auxiliary medical decision-making system, deploys a 24GB NVIDIA RTX3090 GPU as a computing and processing device, and uses Pytorch to implement the processing flow of the method of the present invention. The MIMIC-III dataset is used to verify the feasibility of the method of the present invention.

[0122] The training was completed on the sepsis patient data set in the MIMIC-III data set, and the cost function obtained from the training was applied to the constraint algorithm (CDT) to learn the safety strategy. The safety strategy was verified using sepsis patients in the offline validation set. The drug dosage of this strategy can be recommended to each patient, and the difference between the drug dosage and the actual drug dosage of the doctor can be statistically analyzed. The final death or survival of each patient is known, and the mortality rate of patients under different drug gaps is statistically analyzed. It is known that if the doctor follows the optimal strategy, the mortality rate is 0. Figures 5 to 8 The method of the present invention is capable of capturing an unsafe state in a sepsis environment. Figures 5 to 8 The distribution of the cost function values ​​learned in four aspects, namely SOFA score, lactate level, mean arterial pressure, and heart rate, are shown respectively. The penalty values ​​are higher in the unsafe area and lower in the safe area, indicating that this method can capture unsafe patient states. Fig. 9 is a schematic diagram of the relationship between the penalty value and the mortality rate in the method of the present invention. Fig. 9 The relationship between the penalty value learned by the cost function and the mortality rate is shown. The constraint penalty value of each action is calculated for the patients in the medical data set, and the patient is counted whether he or she finally dies. It is found that as the penalty increases, the mortality rate increases, indicating that the method of the present invention can learn effective penalties and can punish actions that pose a greater threat of death. To evaluate whether a strategy is optimal, it is to evaluate whether the point where the gap is 0 corresponds to a lower mortality rate. Fig.10 and Fig.11 The comparison diagram of the effect of the method of the present invention and the existing DDPG algorithm is shown. Fig.10 and Fig.11The horizontal axis in the middle represents the difference in drug dosage between the strategy and the doctor's strategy. The drug dosage gap between vasopressors and intravenous infusions is calculated respectively, and then the proportion of mortality under this gap is counted. Focus on the point where the horizontal axis is 0, which represents the mortality value when the gap is 0. If the strategy of the present invention is a perfect strategy, then the mortality rate is lowest when the doctor's strategy is close to the strategy of the present invention. It is found that the method of the present invention can obtain a lower mortality rate, that is, the method of the present invention is closer to the optimal strategy. Fig.10 and Fig.11 It can be seen that the method of the present invention can achieve a lower mortality rate, and is therefore closer to the optimal strategy and safer.

[0123] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0124] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by any technician familiar with the technical field within the technical scope disclosed in the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims. The information disclosed in the background technology section of this article is only intended to deepen the understanding of the overall background technology of the present invention, and should not be regarded as an admission or in any form that the information constitutes prior art known to those skilled in the art.

Claims

1. A method for improving the safety of a medical decision-making support system, characterized in that: include: Step 1, obtaining a medical expert dataset and generating a violation dataset through a self-attention-based generative model, wherein both the medical expert dataset and the violation dataset contain a long historical state sequence of each patient; The self-attention-based generation model includes: an input linear layer, a causal attention layer, an action prediction output linear layer, a reward prediction output linear layer, and a next state prediction output linear layer; wherein, The input linear layer is communicatively connected to the input end of the causal attention layer, and can take the long historical state sequence of the patient from the medical expert dataset as input, and encode the input long historical state sequence to obtain the historical state encoding; The causal attention layer is respectively connected to the action prediction output linear layer, the reward prediction output linear layer and the next state prediction output linear layer, and can receive the historical state code output by the input linear layer, compress and learn the historical state code, and obtain the historical state feature; The action prediction output linear layer can receive the historical state features output by the causal attention layer and predict the action based on the historical state features; The reward prediction output linear layer is communicatively connected to the output end of the action prediction output linear layer, can receive the historical state features output by the causal attention layer and the action prediction output by the action prediction output linear layer, and predict the reward based on the historical state features and the action prediction; The next state prediction output linear layer is communicatively connected to the output end of the action prediction output linear layer, can receive the historical state features output by the causal attention layer and the action prediction output by the action prediction output linear layer, and predict the next state according to the historical state features and the action prediction; The action prediction output linear layer predicts the action based on the historical state features in the following manner, including: The action prediction output linear layer By minimizing the negative log-likelihood loss as the optimization objective, while maximizing the weight Entropy to predict actions ,for: ; in, Indicates taking the minimum value; The action prediction output linear layer representing the predicted action Parameters; represents the average value; Before Time step reward, state and previous The actions at each time step are represented as , , ; represents the medical experts dataset; represents the logarithmic function; represents the output of action prediction; represents the weight parameter; represents the Shannon entropy regularizer to avoid overfitting of action outputs; The reward prediction output linear layer Perform reward prediction and the next state prediction output linear layer The next state prediction is formalized according to the following objectives: ; in, Indicates taking the minimum value; and They represent the next state prediction output linear layer parameters for predicting the next state and the reward prediction output linear layer parameters for predicting the reward respectively; represents the average value; Indicates The state of the time step; Indicates The reward for the time step; express and the action prediction output layer in the Predicted action at time step The union of; represents the medical experts dataset; and They represent the next state prediction output linear layer that predicts the next state and the reward prediction output linear layer that predicts the current reward respectively; In step 1, a violation data set is generated based on a self-attention generation model in the following manner, including: Step 121, taking the maximum reward of the long history sequence of patients in the medical expert dataset as the initial target reward and the initial state As input to the self-attention based generative model, the self-attention based generative model predicts the action in an autoregressive manner ; Step 122: Initial target reward , initial state and predicted actions As input to the self-attention based generative model, the predicted reward and the next state ; Step 123, gradually reduce the target reward ,Right now ,in, Indicates Target reward for repetitions; Indicates +1 target bonus for repetition; Indicates Repeated prediction reward; Step 124, gradually derive the subsequent treatment trajectory changes to obtain the treatment trajectory ,in, is the initial state; Actions that are predicted; Rewards for predictions; is the predicted next state; Step 125, repeat steps 121 to 124, with different Initial state As the input of the self-attention based generative model, we can get Different treatment trajectories , useful Different treatment trajectories Composition violation data set , , Represents treatment trajectory Serial number, represents the number of patients in the violation dataset; Step 2: Construct an inverse constraint reinforcement learning model based on causal shielding self-attention: The causal shielding self-attention model is adopted, with the long history state sequence of a patient in the medical expert data set obtained in step 1 and the long history state sequence of a patient in the violation data set generated in step 1 as input, and the action penalty of the patient as output, to obtain the optimization objective equation of the inverse constraint reinforcement learning model based on causal shielding self-attention. Minimizing the objective equation is to obtain the gradient equation of the cost function that minimizes the constraints on the medical expert data and maximizes the constraints on the violation data, which is: ; in, represents the optimization parameters of the inverse constrained reinforcement learning model based on causal masked self-attention; Indicates is the gradient operator of the parameter; represents the cost function calculation equation; Indicates taking the average value; represents a long historical state sequence of a patient from the violation dataset; represents the medical experts dataset; represents the logarithmic function; represents the cost function for violating the dataset, i.e., the penalty for the patient’s actions; Represents a long historical state sequence of a patient from a medical expert dataset; Indicates violation of the data set; Represent the cost function of the medical expert data; Step 3: Apply the cost function obtained in step 2 to learn the constraint strategy: Through step 2, an inverse constrained reinforcement learning model based on causal masked self-attention is obtained. The inverse constrained reinforcement learning model based on causal masked self-attention is trained to obtain a cost function that can observe the patient's long historical state sequence and capture key states. The cost function is used in the existing constrained reinforcement learning algorithm of the medical decision-making system to constrain optimization of the strategy to obtain a safe medical decision-making system.

2. The method for improving the safety of the auxiliary medical decision-making system according to claim 1, characterized in that: In step 1, the medical expert dataset is obtained by using data from an existing medical dataset or data directly obtained from a hospital.

3. The method for improving the safety of the auxiliary medical decision-making system according to claim 2, characterized in that: In step 1, the medical expert dataset is obtained by using data from an existing medical dataset or data directly obtained from a hospital in the following manner, including: Using data from existing medical datasets or data directly obtained from hospitals, the medical expert dataset is composed of long historical state sequences of multiple patients collected in the following manner: exist time step, the state of a patient is represented as , the drug dosage taken by the hospital doctor for the patient's condition is expressed as And the next state of the patient after receiving the dose of the drug is expressed as and the reward is expressed as ; Repeat this process to obtain the treatment trajectory of a patient That is, the patient's long history state sequence, which is composed of multiple long history state sequences of multiple patients collected by records. , ,in, Represents treatment trajectory Serial number, Represents the number of patients in the medical specialists dataset.

4. The method for improving the safety of a medical decision-making support system according to any one of claims 1 to 3, characterized in that: In step 2, the causal shielding self-attention model is composed of a causal self-attention layer and an attention layer connected in sequence; wherein, The causal self-attention layer can take the patient's long historical state sequence as input. All information before the time step is compressed into historical state features ; The attention layer is communicatively connected to the output end of the causal self-attention layer, and can output the historical state features of the causal self-attention layer. The cost function is the weighted sum of input and output.

5. The method for improving the safety of the auxiliary medical decision-making system according to claim 4, characterized in that: In step 2, the causal self-attention layer uses a linear layer to use matrix multiplication to convert each historical state feature Project to different subspaces to generate query values , key value and cost value ,in, represents the time step; Represents the total length of the patient's long history state sequence; represents the query value, which is the causal self-attention layer in the The information that the time step hopes to obtain from other locations; Indicates the key value, which is The information characteristics actually contained in the time step; The causal self-attention layer converts the query value into and key value Obtain weight representation through matrix calculation ,in, express The weight of the time step, express The cost of the time step; Represents the total length of the patient's long history state sequence; The cost function is obtained by taking the weighted sum of the costs at each time step and expressing the final cost at each time step. , , where the meaning of each parameter is: represents the sum function; represents the time step; The total length of the sequence representing the patient's long history state sequence; express The weight of the time step; express The cost of a time step.

6. A processing device, characterized in that: include: at least one memory for storing one or more programs; At least one processor can execute one or more programs stored in the memory, and when the one or more programs are executed by the processor, the processor can implement the method described in any one of claims 1 to 5.

7. A readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 5 can be implemented.