Moxibustion decision-making method and system based on Bert pre-training and deep reinforcement learning

Abstract

The invention discloses a moxibustion decision-making method and system based on Bert pre-training and deep reinforcement learning, and belongs to the artificial intelligence technology and the medical industry, and the method comprises the steps: 1, data collection: collecting data related to a moxibustion therapy, including case data, medical literature and moxibustion operation records; carrying out data cleaning and preprocessing on the collected data, removing noise, processing missing values and carrying out standardized data processing; 2, according to the user data information and a pre-trained Bert network model, state representation of moxibustion diagnosis and treatment related information is carried out; and step 3, based on the obtained state representation, inputting the state representation into a deep reinforcement learning network model, and performing decision learning of the intelligent agent. The system serves the patient, realizes autonomy, intelligentization and individuation of moxibustion diagnosis and treatment, and better promotes the development of moxibustion diagnosis and treatment.

Description

Technical Field

[0001] The present invention belongs to artificial intelligence technology and medical industry, and in particular to a moxibustion decision-making method and system based on BERT pre-training and deep reinforcement learning. Background Art

[0002] With the in-depth development of artificial intelligence technology in the medical field and the invention of equipment such as robotic arms and sensor devices, moxibustion diagnosis and treatment technology has been well developed. Moxibustion robots have gradually entered thousands of households. Tasks that originally required moxibustion experts to complete can gradually be completed by robots, reducing the difficulty of diagnosis and treatment operations.

[0003] Among them, application number CN202010302192.7 proposes a traditional Chinese medicine moxibustion system and method, which simulates the moxibustion operation process through sensors and robotic arms, and constructs a Markov model for moxibustion decision-making based on the temperature data of the skin acupoints under moxibustion within a certain period of time.

[0004] Among them, the application with application number CN202210356305.0 provides a method and system for optimizing dynamic diagnosis and treatment plans of traditional Chinese medicine based on deep reinforcement learning, which combines the reinforcement learning model to train the diagnosis and treatment optimization model to obtain the best patient diagnosis and treatment plan.

[0005] Traditional moxibustion requires experts to hold moxa sticks and repeatedly adjust the distance between the moxa sticks and the skin of the acupuncture points, which is very inefficient. With the development of technology, new moxibustion tools and moxibustion robots have emerged. Although they are easy to operate and relatively safe, they cannot "prescribe the right medicine" for the acupuncture points, so the treatment effect is not good. Therefore, according to the individual case of the patient, personalized diagnosis and treatment plans are recommended, and combined with reinforcement learning, the moxibustion robot learns the best strategy for moxibustion, which can greatly improve the treatment effect. Summary of the invention

[0006] The present invention uses the Bert model to perform intent recognition and entity recognition on moxibustion-related questions input by users, constructs a state representation of moxibustion diagnosis and treatment, and then uses the deep reinforcement learning algorithm DQN (Deep Q Network) to perform agent decision training, so that the moxibustion robot can learn the best strategy for moxibustion and provide personalized moxibustion diagnosis and treatment plans. It is different from the above application in design ideas and functional modules.

[0007] Specifically, the present invention provides the following technical solutions:

[0008] A moxibustion decision-making method based on Bert pre-training and deep reinforcement learning includes the following steps:

[0009] Step 1: Data collection: collect data related to moxibustion therapy, including case data, medical literature, and moxibustion operation records; clean and preprocess the collected data to remove noise, process missing values, and standardize data processing;

[0010] Step 2: Obtain the state representation of moxibustion treatment related information based on user data information and the pre-trained Bert network model;

[0011] Step 3: Based on the acquired state representation, it is input into the deep reinforcement learning network model to perform decision learning of the intelligent agent.

[0012] The present invention also proposes a moxibustion decision system based on Bert pre-training and deep reinforcement learning, comprising:

[0013] Moxibustion data collection module: used to obtain moxibustion diagnosis and treatment data;

[0014] Bert model training module: used to extract effective features for subsequent model training and decision-making;

[0015] Moxibustion diagnosis and treatment decision module: used to continuously interact with the environment and adjust parameters, allowing the intelligent agent to learn the optimal moxibustion diagnosis and treatment decision.

[0016] The present invention has the following beneficial effects:

[0017] The above-mentioned moxibustion method and system based on Bert model and deep reinforcement learning can calculate the diagnosis and treatment plan required by the user according to the patient's actual case information, put the diagnosis and treatment plan into DQN for learning feedback, let the intelligent agent obtain the corresponding diagnosis and treatment plan, and evaluate and feedback the final diagnosis and treatment plan, ultimately serving the patient himself, realizing the autonomy, intelligence and personalization of moxibustion diagnosis and treatment, and better promoting the development of moxibustion diagnosis and treatment. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 is a flow chart of a method in an embodiment of the present invention;

[0019] Figure 2 It is a structural diagram of a moxibustion decision-making system based on Bert pre-training and deep reinforcement learning in an embodiment of the present invention.

[0020] The reference numerals in the figure are: moxibustion data collection module 100, Bert model training module 200, and moxibustion diagnosis and treatment decision module 300. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical scheme and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in each embodiment of the present invention described below can be combined with each other as long as they do not conflict with each other. To achieve the above-mentioned purpose, the present invention adopts the following technical scheme.

[0022] Figure 1 This is a flowchart of moxibustion decision-making based on Bert pre-training and deep reinforcement learning. The method includes:

[0023] Step 1: Data collection: collect data related to moxibustion therapy and convert the collected data into an input form acceptable to the Bert model; specifically, it includes:

[0024] S1: Obtain the user's clinical moxibustion diagnosis and treatment data;

[0025] The collected data include open source data sets, records of diagnosis and treatment conversations between doctors and patients, and clinical data related to moxibustion: including medical records, symptom descriptions, moxibustion plans, etc.

[0026] S11: Based on the information obtained in S1, convert the text data into an input format acceptable to the Bert model. Preprocess the text data such as user needs, symptom descriptions, and moxibustion plans, use the word segmentation tool jieba library to segment the words, and then tokenize and encode the word segmentation results and convert them into the input format of the BERT model, that is, convert each sentence into a vector sequence and add a special identifier "[CLS]" before each paragraph of text.

[0027] Step 2: Based on the user data information and the pre-trained Bert network model, obtain the state representation of moxibustion diagnosis and treatment related information; specifically including:

[0028] S2: Use the pre-trained Bert model for feature extraction, and input the pre-processed text data into the Bert model to obtain the output of the BERT model, including the word vector representation of each word and the sentence-level representation. Then use a small amount of labeled data for fine-tuning to adapt to the intent recognition and entity recognition tasks of moxibustion diagnosis and treatment.

[0029] S21: After obtaining the representation vector of the text, the Softmax classifier is used to calculate the probability distribution of the text belonging to each category, and the classification loss is calculated based on the probability distribution. Finally, back propagation is performed based on the loss to update the parameters of the entire model. The classification loss is calculated using the cross entropy loss function. The specific formula is as follows:

[0030] ,

[0031] In the formula, M represents the types of all text classifications, represents the probability that the classifier predicts that text o is of category c. When category c is the true category of text o, The value is 1, otherwise The value of is 0.

[0032] S22: Fuse the extracted vectors with the output of the Bert model to construct a complete state representation. Use concatenation, weighted averaging, and other methods to fuse different features to form a comprehensive state representation. The state representation of moxibustion diagnosis and treatment related information includes information such as user needs, condition description, and the current moxibustion plan and treatment progress. The details are as follows:

[0033] User needs: including the user's purpose of moxibustion, symptom description, pain level, etc. This information is used for intent recognition and entity recognition through the Bert model to convert the user's natural language input into a representation that the machine can understand.

[0034] Condition description: including the patient's medical history, past treatment, disease progression, etc. This information is encoded through the Bert model, converting text data into vector representation.

[0035] Current moxibustion scheme: including currently selected moxibustion points, moxibustion time, temperature, etc. This information is represented by discrete one-hot encoding.

[0036] Treatment progress: including the degree of improvement of the condition and patient feedback. This information is evaluated by setting three indicators, namely (1) Disease improvement index: by comparing the improvement of patients' symptoms before and after treatment to evaluate the treatment progress, a symptom improvement degree scoring system is set up, using a score of 0-10, 0 for no improvement and 10 for complete improvement. According to the changes in patients' symptoms, the corresponding score is given, and then the average score is calculated to evaluate the treatment progress. (2) Pain level assessment: For the improvement of pain symptoms, the pain visual analog scale (VAS) tool is used to ask patients to give a score of 0-10 based on their feelings, 0 for no pain and 10 for the most severe pain. The treatment progress is evaluated by comparing the pain scores before and after treatment. (3) Patient satisfaction assessment: by designing a satisfaction questionnaire, including patients' evaluation of treatment effect, pain relief, quality of life improvement, etc., the patient satisfaction score is calculated to evaluate the treatment progress.

[0037] Step 3: Based on the acquired state representation, input it into the deep reinforcement learning network model to conduct decision-making learning of the intelligent agent. Specifically including: S3: Establishing a moxibustion diagnosis and treatment decision-making system based on deep reinforcement learning:

[0038] In the process of moxibustion decision-making, the state representation based on the output of the Bert model is used as the input state of the agent (moxibustion robot) decision. According to the application scenario and needs of the agent, the action space of the agent is defined. The action space includes selecting different moxibustion points, adjusting moxibustion time and temperature, etc. Each action is encoded as a discrete action label. Taking the action space definition of a thermosensitive moxibustion as an example, the corresponding label data is designed: actions = {

[0039] "Moxibustion acupoint selection": ["Baihui acupoint", "Fengchi acupoint", "Dazhui acupoint", "Yongquan acupoint", "Taichong acupoint"],

[0040] "Moxibustion time adjustment": ["5 minutes", "10 minutes", "15 minutes", "20 minutes", "25 minutes"],

[0041] "Moxibustion temperature adjustment": ["Low temperature", "Medium-low temperature", "Medium temperature", "Medium-high temperature", "High temperature"]

[0042] }.

[0043] Through interaction with the environment, the robot selects the best action according to the current state to optimize the moxibustion strategy. According to the current state and the selected action, the state transfer is simulated and the reward function is defined. The reward function is comprehensively evaluated based on three indicators: the degree of improvement of the condition, patient feedback, and doctor evaluation, to guide the reinforcement learning model to learn a good decision-making strategy. The value range of the degree of improvement index is 0 to 1, where 0 means that the condition has not improved and 1 means that the condition has completely improved, and the weight is set to 0.4. The value range of the patient feedback index is 0 to 1, where 0 means that the patient is not satisfied with the moxibustion effect and 1 means that the patient is very satisfied with the moxibustion effect, and the weight is set to 0.3; the value range of the doctor evaluation index is 0 to 1, where 0 means that the doctor thinks the moxibustion effect is very poor and 1 means that the doctor thinks the moxibustion effect is very good, and the weight index is set to 0.3.

[0044] Using the deep reinforcement learning algorithm: DQN, a neural network model based on moxibustion decision-making is established. The input of the model is the state representation, and the output is the corresponding action of the agent. Using the convolutional neural network (CNN) as the network structure of DQN, the model is trained by the reinforcement learning algorithm using the existing state transition data and reward function, and experience replay is used to improve the training effect and stability. During the training process, the model will gradually optimize the decision-making strategy by constantly trying different actions and providing feedback based on the reward function. In practical applications, according to the current state input into the trained reinforcement learning model, the model will output an action, that is, the decision that the moxibustion diagnosis and treatment system should take. According to the action output by the model, the system can allow the agent to perform corresponding moxibustion operations or adjustments.

[0045] In addition, a moxibustion diagnosis and treatment system based on the Bert model and deep reinforcement learning is also provided.

[0046] Figure 2 The diagram is a schematic diagram of the structure of a moxibustion decision system based on Bert pre-training and deep reinforcement learning in an implementation example. The system includes a moxibustion data acquisition module 100, a Bert model training module 200, and a moxibustion diagnosis and treatment decision module 300.

[0047] The moxibustion data collection module 100 is used to obtain moxibustion diagnosis and treatment data.

[0048] The user logs into the system, fills in the user's personal CRF form, collects relevant data of the patient, including the patient's basic information (name, age, gender, medical history, etc.); symptom description (main symptoms, pain description, other symptoms, etc.), stores the above information in text form, and uploads it to the cloud platform. The collected data set is preprocessed by cleaning, denoising, and normalization to ensure the accuracy and consistency of the data.

[0049] The Bert model training module 200 is used to extract effective features for subsequent model training and decision making.

[0050] The collected data is encoded and processed through the pre-trained Bert model to extract the main features, and the moxibustion effect is predicted according to the patient's characteristics. According to the results predicted by the model, the specific operation plan of moxibustion is determined, such as selecting acupoints, moxibustion methods, degree of heat, etc., and it is generated as the output data of the Bert model and input into the next module as the initial state representation for intelligent agent decision-making.

[0051] The moxibustion diagnosis and treatment decision module 300 is used to train the intelligent agent (moxibustion robot) to learn relevant decision-making operations.

[0052] By accepting the predicted state in the Bert model training module 200 and taking it as the initial state in the intelligent agent's decision-making process, combined with the pre-set action space and reward function, and inputting the network model training intelligent agent based on deep reinforcement learning, the intelligent agent will output an action, that is, the decision that the moxibustion robot should take, and by constantly interacting with the environment and updating model parameters, the moxibustion robot can learn the best strategy for moxibustion and improve personalized moxibustion treatment plans.

Claims

1. A moxibustion decision-making method based on Bert pre-training and deep reinforcement learning, characterized in that: The steps include: Step 1: Data collection, collect data related to moxibustion therapy and convert the collected data into an input form acceptable to the Bert model; Step 2: Obtain the state representation of moxibustion treatment related information based on user data information and the pre-trained Bert network model; Step 3: Based on the acquired state representation, it is input into the deep reinforcement learning network model to perform decision learning of the intelligent agent.

2. A moxibustion decision-making method based on Bert pre-training and deep reinforcement learning according to claim 1, characterized in that: In step 1, the collected data include open source data sets, specifically: records of medical conversations between doctors and patients, and clinical data related to moxibustion: including user needs, medical records, symptom descriptions, and moxibustion plans.

3. A moxibustion decision-making method based on Bert pre-training and deep reinforcement learning according to claim 2, characterized in that: In step 1, the preprocessing is to use the word segmentation tool jieba library to perform word segmentation, and then tokenize and encode the word segmentation results and convert them into the input format of the Bert model, that is, convert each sentence into a vector sequence and add the identifier "[CLS]" before each paragraph of text.

4. A moxibustion decision-making method based on Bert pre-training and deep reinforcement learning according to claim 3, characterized in that: Step 2 specifically includes: using the pre-trained Bert model to extract data features and obtain the text representation vector; classifying the text representation vector through Softmax to extract the corresponding classification features; fusing the extracted vector and classification features with the output of the Bert model to construct a complete state representation.

5. A moxibustion decision-making method based on Bert pre-training and deep reinforcement learning according to claim 4, characterized in that: Step 2 is further refined as follows: S2: Use the pre-trained Bert model for feature extraction, and input the pre-processed text data into the Bert model to obtain the output of the Bert model, including the word vector representation and sentence-level representation of each word, and then use the labeled data for fine-tuning to adapt to the intent recognition and entity recognition tasks of moxibustion diagnosis and treatment; S21: After obtaining the representation vector of the text, the Softmax classifier is used to calculate the probability distribution of the text belonging to each category, and the classification loss Loss is calculated based on the probability distribution. Finally, back propagation is performed based on the loss to update the parameters of the entire model, and the classification loss is calculated using the cross entropy loss function. The specific formula is as follows: ， In the formula, M represents the types of all text classifications, represents the probability that the classifier predicts that text o is of category c, when category c is the true category of text o. The value is 1, otherwise The value of is 0; S22: The extracted vectors and classification features are integrated with the output of the Bert model to construct a complete state representation. Different features are integrated using splicing and weighted averaging to form a comprehensive state representation. The state representation of information related to moxibustion diagnosis and treatment includes user needs, disease description information, and the current moxibustion plan and treatment progress.

6. A moxibustion decision-making method based on Bert pre-training and deep reinforcement learning according to claim 5, characterized in that: User needs, condition description, current moxibustion program and treatment progress are as follows: User needs include the user's purpose of moxibustion, symptom description, and pain level. This information is used through the Bert model for intent recognition and entity recognition, converting the user's natural language input into representations that the machine can understand. The description of the condition includes the patient's medical history, past treatments, and the progression of the condition. This information is encoded using the Bert model, which converts text data into vector representations. The current moxibustion plan includes the currently selected moxibustion points, moxibustion time, and temperature. These information are represented by discrete one-hot coding. Treatment progress includes the degree of improvement of the condition and patient feedback. This information is evaluated by setting three indicators: (1) Disease improvement indicator: The treatment progress is evaluated by comparing the improvement of patients' symptoms before and after treatment. A symptom improvement scoring system is set up, using a score of 0-10, with 0 indicating no improvement and 10 indicating complete improvement. According to the changes in patients' symptoms, corresponding scores are given, and then the average score is calculated to evaluate the treatment progress. (2) Pain level assessment: For the improvement of pain symptoms, the pain visual analogue scale (VAS) tool is used to ask patients to give a score of 0-10 based on their feelings, with 0 indicating no pain and 10 indicating the most severe pain. The treatment progress is evaluated by comparing the pain scores before and after treatment. (3) Patient satisfaction assessment: A satisfaction questionnaire is designed, including patients' evaluation of treatment effects, pain relief, and improvement in quality of life. The patient satisfaction score is calculated to evaluate the treatment progress.

7. A moxibustion decision-making method based on Bert pre-training and deep reinforcement learning according to claim 6, characterized in that: In step 3, in the moxibustion decision-making process, the state representation based on the output of the Bert model is used as the input state of the agent's decision. According to the application scenario and requirements of the agent, the action space of the agent is defined. The action space includes selecting different moxibustion points, adjusting the moxibustion time and temperature, and encoding each action as a discrete action label. Through interaction with the environment, the agent selects the best action according to the current state to optimize the moxibustion strategy; according to the current state and the selected action, the state transfer is simulated and a reward function is defined. The reward function performs a comprehensive evaluation based on three indicators: the degree of improvement of the condition, patient feedback and doctor's evaluation, so as to guide the reinforcement learning model to learn a good decision-making strategy. The value range of the indicator of the degree of improvement of the condition is 0 to 1, where 0 means that the condition has not improved and 1 means that the condition has completely improved. The weight is set to 0.

4. The value range of the patient feedback indicator is 0 to 1, where 0 means that the patient is not satisfied with the moxibustion effect and 1 means that the patient is very satisfied with the moxibustion effect. The weight is set to 0.3; the value range of the doctor's evaluation indicator is 0 to 1, where 0 means that the doctor thinks the moxibustion effect is very poor and 1 means that the doctor thinks the moxibustion effect is very good. The weight indicator is set to 0.

3.

8. A moxibustion decision-making method based on Bert pre-training and deep reinforcement learning according to claim 7, characterized in that: The intelligent agent uses the deep reinforcement learning algorithm DQN to establish a neural network model based on moxibustion decision-making. The input of the model is the state representation, and the output is the corresponding action of the intelligent agent. A convolutional neural network is used as the network structure of DQN. The existing state transition data and reward function are used to train the model through the reinforcement learning algorithm. Experience replay is used to improve the training effect and stability. During the training process, the model will continuously try different actions and provide feedback based on the reward function to gradually optimize the decision-making strategy. In actual applications, the current state is input into the trained reinforcement learning model, and the model outputs an action, that is, the decision that the moxibustion diagnosis and treatment system should take. According to the action output by the model, the intelligent agent can perform corresponding moxibustion operations or adjustments.

9. A moxibustion decision system based on Bert pre-training and deep reinforcement learning, characterized in that: include: Moxibustion data collection module: used to obtain moxibustion diagnosis and treatment data; Bert model training module: used to extract effective features for subsequent model training and decision-making; Moxibustion diagnosis and treatment decision module: used to continuously interact with the environment and adjust parameters, allowing the intelligent agent to learn the optimal moxibustion diagnosis and treatment decision.

Images