Traditional Chinese medicine multi-formula collaborative optimization method based on monte carlo tree search and deep reinforcement learning

Abstract

The present application relates to the field of traditional Chinese medicine multi-formula synergistic optimization, and particularly relates to a traditional Chinese medicine multi-formula synergistic optimization method based on Monte Carlo tree search and deep reinforcement learning. The method comprises the following steps: obtaining a disease target, patient characteristics, a number of stages, a search budget, a compatibility contraindication rule library, a policy network and a value network, and generating an optimization configuration package; constructing a stage state according to the stages and generating a stage state package; performing Monte Carlo tree search, with the policy network outputting an action prior probability, the value network outputting a state value, and a legal action set being generated according to the compatibility contraindication rule library; simulating a termination state by using a greedy rolling strategy, calculating an efficacy, safety, cost and synergistic score and generating a total reward, returning and updating a state value, and outputting a stage formula package; and finally, summarizing a multi-stage formula set and performing safety and dose adjustment. The present application realizes dynamic synergistic optimization and self-learning evolution of multi-stage traditional Chinese medicine formulas, and improves the scientificity and interpretability of combination screening.

Description

Technical Field

[0001] This invention relates to the field of synergistic optimization of multiple prescriptions of traditional Chinese medicine, and in particular to a method for synergistic optimization of multiple prescriptions of traditional Chinese medicine based on Monte Carlo tree search and deep reinforcement learning. Background Technology

[0002] In the field of collaborative optimization of multiple prescriptions in traditional Chinese medicine, existing solutions for generating multi-stage prescription sets based on disease objectives, patient characteristics, and stage number typically construct candidate prescriptions from drug sets, perform conflict verification using a rule base for incompatibilities, and score and rank the candidate prescriptions under search budget constraints to form stage prescription packages. These stage prescription packages are then aggregated to obtain the multi-stage prescription set. However, this approach suffers from limitations such as difficulty in uniformly expressing stage states, difficulty in incorporating inter-stage collaborative relationships into the same evaluation chain, and insufficient coupling between the rule base for incompatibilities and the candidate generation chain. Existing methods commonly generate drug sets separately for different stages and output stage prescription packages independently. When the stage number is greater than the first stage, there is a lack of a unified approach for writing the previous stage prescription code into the stage state, resulting in a lack of stable stage state input for subsequent collaborative scoring calculations of the multi-stage prescription set. Furthermore, existing methods often focus on a single dimension of efficacy or safety scores on the scoring side, while cost and collaborative scores are often placed in post-screening stages or calculated separately, making it difficult to form a total reward chain consistent with the stage states. This, in turn, affects the statistical consistency of the number of back-up update visits and state-action values. On the search side, existing solutions often struggle to organize the complete Monte Carlo tree search process under the constraints of stage state packages. The fusion of the policy network's output action prior probabilities and the value network's output state values ​​lacks a unified in-tree statistical structure. The updates to the relationships between state-action values, access counts, and action prior probabilities lack consistent recording granularity, making it difficult for the search tree statistics package to support continuous calls to generate termination states using the greedy rolling strategy simulation. Simultaneously, the mismatch rule base is often used as a post-event validation item in the candidate action generation stage. The validation paths between adding, removing, and replacing actions and the set of legal actions lack synchronization constraints during the tree expansion stage, easily introducing duplicate invalid expansions or generating conflict handling branches that are difficult to trace. For the joint processing of disease objectives, patient characteristics, number of stages, search budget, mismatch rule base, policy network, and value network, existing technologies generally lack a seamless collection-validation-search-statistics-summarization process in terms of configuration item archiving, version consistency recording, abnormal input handling, and cross-stage connections. This makes it difficult to maintain consistency in the input-output correspondence between stage state packages, search tree statistics packages, and multi-stage formula sets. Summary of the Invention

[0003] To address the aforementioned technical problems, this invention provides a method for synergistic optimization of multiple traditional Chinese medicine formulas based on Monte Carlo tree search and deep reinforcement learning, comprising:

[0004] S100: Obtain disease objectives, patient characteristics, number of stages, search budget, incompatibilities rule base, strategy network and value network, and generate an optimized configuration package;

[0005] S200: Based on the optimized configuration package, construct the stage status in stages. The stage status includes drug set, patient characteristics, disease goal and stage identifier. When the stage number is greater than the first stage, write the formula code of the previous stage into the stage status to generate the stage status package.

[0006] S300: Perform Monte Carlo tree search based on the stage state package. The policy network outputs the prior probability of the action and the value network outputs the state value. Calculate the upper confidence bound tree index selection based on the state-action value, number of visits, and prior probability of the action. Generate a legal action set expansion by verifying the addition, removal, and replacement of actions based on the matching taboo rule base, and generate a search tree statistics package.

[0007] S400: Based on the search tree statistical package, a greedy rolling strategy with exploration ratio parameters is used to simulate and generate the termination state. The efficacy score, safety score, cost score and synergy score are calculated and combined in a multiplicative manner to generate the total reward. The number of visits and the state-action value are updated and the optimal action sequence of the stage is extracted to generate the stage formula package.

[0008] S500: Based on the phased formulation package, summarize the multi-stage formulation set, perform safety review and dosage ratio adjustment, and generate a multi-stage formulation set.

[0009] Furthermore, the preceding stage formulation code includes the drug set code, efficacy embedding code, and stage sequence code of the preceding stage formulation, wherein the efficacy embedding code is obtained by retrieving the efficacy embedding library according to the drug identifier.

[0010] Furthermore, the set of legal actions is obtained by verifying the candidate action set through a compatibility rule base. The verification includes: applying the candidate action to the drug set to generate a candidate formula, performing a compatibility conflict detection on the candidate formula, and writing the candidate action with a negative compatibility conflict detection result into the set of legal actions.

[0011] Furthermore, the process of the policy network outputting the action prior probability includes: constructing a state-action encoding vector for the stage state and candidate actions, wherein the state-action encoding vector includes the graph embedding of the current formula, the action type encoding and the target drug embedding, and performing normalized probability mapping on the state-action encoding vector to generate the action prior probability.

[0012] Furthermore, the process of the value network outputting state value includes: constructing a state encoding vector for the stage state, and generating a value item and a confidence item based on the state encoding vector, wherein the state value is obtained by multiplying the value item and the confidence item.

[0013] Furthermore, the greedy rolling strategy includes: selecting the action with the highest prior probability at a ratio minus the exploration ratio parameter, and randomly selecting an action at the ratio of the exploration ratio parameter.

[0014] Furthermore, the feedback update includes: incrementing the access count by one, and updating the state-action value by dividing the difference between the total reward and the original state-action value by the updated access count.

[0015] Furthermore, the efficacy score is generated by the matching probability output of the efficacy prediction model for the disease target and the drug set. The calculation of the efficacy score includes: performing a log-odds transformation on the matching probability, calculating the deviation penalty term between the matching probability and the target probability, performing a weighted fusion of the log-odds transformation result and the deviation penalty term, and multiplying the fusion result by the confidence weight to obtain the efficacy score.

[0016] Furthermore, the calculation of the collaborative score includes: extracting efficacy vectors from two-stage formulations within a multi-stage formulation set and calculating cosine similarity; calculating the intersection-union ratio (IUR) of the drug sets of the two-stage formulations; calculating complementarity and redundancy terms based on cosine similarity and IUR; and generating a collaborative score by subtracting redundancy terms from complementarity terms.

[0017] Furthermore, the policy network and the value network are obtained through self-play training, which includes: performing Monte Carlo tree search on the training samples to obtain a training dataset consisting of the state, access number normalization policy, and total reward; calculating the policy loss, value loss, and entropy regularization loss based on the training dataset, and performing a weighted summation of the three types of losses to obtain the training loss; and updating the policy network parameters and value network parameters according to the training loss.

[0018] The key innovations of this invention include:

[0019] (1) Based on the optimized configuration package, the stage status is constructed in stages, and when the stage number is greater than the first stage, the formula code of the previous stage is written into the stage status to form a stage status package containing drug set, patient characteristics, disease target and stage identifier, so as to realize the unified organization and traceable connection of stage status and previous stage information.

[0020] (2) Based on the stage state package, perform Monte Carlo tree search, introduce the prior probability of the action output by the policy network and the state value output by the value network, and select the upper confidence bound tree index according to the state-action value, access count and action prior probability. At the same time, verify the addition, removal and replacement actions according to the matching taboo rule base to generate a legal action set expansion, forming a verifiable search tree statistical package.

[0021] (3) Based on the search tree statistical package, simulate the termination state according to the greedy rolling strategy of the exploration ratio parameter, calculate the efficacy score, safety score, cost score and synergy score and generate the total reward by multiplication, return the updated access number and state-action value and extract the optimal action sequence of the stage to generate the stage formula package, so that the stage formula package establishes a closed-loop correspondence with the total reward and the statistical update in the tree, and is used for the summary processing of multi-stage formula sets.

[0022] The following are its main beneficial effects:

[0023] (1) In view of the common problems in the background technology that it is difficult to uniformly express the stage status and that the previous stage formula code is missing when the stage number is greater than the first stage, the stage status package encapsulates the drug set, patient characteristics, disease target and stage identifier in the same carrier, and writes the previous stage formula code as a component field of the stage status, so as to realize the traceable connection between the stages of the multi-stage formula, and so that the Monte Carlo tree search is performed based on the stage status package in the future, which is suitable for the operation scenario that needs to form a multi-stage formula set according to the stage.

[0024] (2) In view of the problem that the mismatch taboo rule base and candidate generation link are not sufficiently coupled and the statistical bearing structure within the tree is not uniform, which makes it difficult to continuously call the search tree statistical package, the output of the policy network and the value network are fused with the state-action value, access count and action prior probability in the same upper confidence bound tree index selection link. In the tree expansion stage, the legality of adding, removing and replacing actions is verified according to the mismatch taboo rule base to generate a legal action set expansion, so that the search process forms a verifiable search tree statistical package under the stage state package constraint. It is suitable for operation scenarios with mismatch taboo constraints and need to expand actions within the search budget.

[0025] (3) In response to the common problem in the background technology that the scoring side is mostly focused on the single dimension of efficacy score or safety score, and the cost score and synergy score are placed in the post-screening, it is difficult to form a total reward link consistent with the stage state and affect the consistency of the number of back-up update accesses and the state-action value statistics. When the greedy rolling strategy simulates the generation of the termination state, the efficacy score, safety score, cost score and synergy score are calculated simultaneously and the total reward is generated by multiplication. The total reward is used to back-up update accesses and state-action values ​​and to extract the optimal action sequence of the stage to generate the stage recipe package. This makes the stage recipe package and the search tree statistics package form a consistent closed-loop update path. It is suitable for operation scenarios that need to consider multi-dimensional scores and summarize multi-stage recipe sets at the same time. Attached Figure Description

[0026] Figure 1A flowchart illustrating a method for synergistic optimization of multiple traditional Chinese medicine formulas based on Monte Carlo tree search and deep reinforcement learning, provided for embodiments of this application;

[0027] Figure 2 This application provides a Monte Carlo tree search recipe optimization system architecture diagram.

[0028] Figure 3 Example diagram of a Monte Carlo tree search provided for embodiments of this application;

[0029] Figure 4 A three-stage formulation efficacy vector radar chart provided for embodiments of this application. Detailed Implementation

[0030] Figure 2 The system architecture diagram of the MCTS formulation optimization system provided in the embodiments of this application is as follows: Figure 2 As shown, the system architecture is divided into an interaction layer, a search layer, an evaluation layer, and a knowledge layer from top to bottom, which together support the collaborative optimization of multi-stage formulas.

[0031] The interaction layer is used to receive and parse the disease goals, patient individual characteristics (such as physical condition, syndrome type, comorbidities) and optimization preferences (such as focusing on efficacy, cost or safety) input by the user, and encapsulate them into a standardized optimization configuration package to provide goals and constraints for the subsequent optimization process.

[0032] The search layer is the core decision engine of the system, employing a Monte Carlo tree search algorithm framework. Based on the current stage state, it iteratively executes four steps of selection, expansion, simulation, and backtracking within an action space consisting of drug operations such as adding, removing, and replacing drugs. It dynamically constructs and explores a prescription decision tree, aiming to find the optimal sequence of actions from the initial state to meeting termination conditions (such as reaching the stage number or budget exhaustion).

[0033] The evaluation layer provides intelligent assessment for the search layer's decision-making. Its core is a deep reinforcement learning model composed of a policy network and a value network. The policy network receives state codes and outputs the prior probabilities of each candidate action, guiding the search towards high-potential directions. The value network performs an overall value assessment of the states, used to quickly predict future returns during simulation. At the end of the simulation, this layer invokes a multi-objective reward function to comprehensively quantify and score the simulated formulation based on four dimensions: efficacy, safety, cost, and inter-stage synergy. This reward is then fed back to update the statistical data of the search tree nodes.

[0034] The knowledge layer, which forms the foundation of the system's operation, provides domain knowledge and data support for the aforementioned layers. It includes a structured drug database (containing information on properties, flavors, meridian tropism, efficacy, and costs), a base of drug compatibility rules (based on theories such as the Eighteen Incompatibilities and Nineteen Antagonisms), and a trained efficacy prediction model. This knowledge layer is not only used to construct states and verify the legality of actions (such as avoiding incompatible combinations), but also directly participates in efficacy prediction and reward calculation during the simulation process.

[0035] The aforementioned four-layer architecture works collaboratively to deeply integrate knowledge from the field of traditional Chinese medicine, modern optimization algorithms, and deep learning models, achieving an automated and intelligent process from multi-objective input to multi-stage optimized formula output. The following section will detail the specific implementation methods of each step based on this system architecture.

[0036] Reference Figure 1 This is a flowchart illustrating a method for synergistic optimization of multiple traditional Chinese medicine formulas based on Monte Carlo tree search and deep reinforcement learning, provided by an embodiment of the present invention. The flowchart may include at least steps S100-S500:

[0037] S100: Obtain disease objectives, patient characteristics, number of stages, search budget, incompatibilities rule base, strategy network and value network, and generate an optimized configuration package;

[0038] S200: Based on the optimized configuration package, construct the stage status in stages. The stage status includes drug set, patient characteristics, disease goal and stage identifier. When the stage number is greater than the first stage, write the formula code of the previous stage into the stage status to generate the stage status package.

[0039] S300: Perform Monte Carlo tree search based on the stage state package. The policy network outputs the prior probability of the action and the value network outputs the state value. Calculate the upper confidence bound tree index selection based on the state-action value, number of visits, and prior probability of the action. Generate a legal action set expansion by verifying the addition, removal, and replacement of actions based on the matching taboo rule base, and generate a search tree statistics package.

[0040] S400: Based on the search tree statistical package, a greedy rolling strategy with exploration ratio parameters is used to simulate and generate the termination state. The efficacy score, safety score, cost score and synergy score are calculated and combined in a multiplicative manner to generate the total reward. The number of visits and the state-action value are updated and the optimal action sequence of the stage is extracted to generate the stage formula package.

[0041] S500: Based on the phased formulation package, summarize the multi-stage formulation set, perform safety review and dosage ratio adjustment, and generate a multi-stage formulation set.

[0042] S100: Obtain disease objectives, patient characteristics, number of stages, search budget, incompatibilities rule base, strategy network and value network, and generate an optimized configuration package;

[0043] Specifically, S100 is triggered by the interaction module upon receiving a request for collaborative optimization of multiple TCM formulas. This request originates from the prescription decision terminal of a hospital information system, the prescription workstation of a telemedicine platform, or the batch task scheduler of a research platform. The interaction module performs field parsing and consistency verification on the request payload, extracting disease objectives and patient characteristics, and reads the stage number and search budget from the request metadata. The disease objective is a structured target description oriented towards the patient, including diagnostic codes, syndrome labels, chief symptom labels, target efficacy labels, and prohibited efficacy labels. The patient characteristics are a structured feature set oriented towards individual differences, including age stratification labels, constitution labels, allergy history labels, complication labels, previous medication labels, and laboratory test summary fields. The stage number is a parameter representing the quantity of treatment stage identifiers. The search budget is a resource constraint parameter for the search process, consisting of an upper limit for the number of iterations, an upper limit for node expansion, and an upper limit for the number of simulations, all registered in the same unit system. In scenarios where there are missing tests or conflicts between the disease target and the patient characteristics, S100 performs missing test marking and conflict resolution processing. The missing test marking includes a field-level missing test flag and a source credibility flag. Conflict resolution adopts a source priority rule and records the pre-conflict value, post-conflict value, and resolution rule identifier. The missing test marking and the conflict resolution record are written together with the disease target and the patient characteristics into the subsequently generated optimization configuration package. The interaction module simultaneously generates a session identifier and a timestamp and binds them to the optimization configuration package, thereby associating and archiving the log records of an optimization request with the subsequent stage state construction and search process.

[0044] In the S100 processing, the drug incompatibilities rule base is loaded and version-fixed by the knowledge module. The rule base is a structured set of rule entries, each containing drug-pair incompatibilities, drug-syndrome incompatibilities, drug-population characteristic incompatibilities, dose-related toxicity constraint fragments, and alternative suggestion fragments. Drug-pair incompatibilities cover the mapping of entries for the "Eighteen Incompatibilities" and "Nineteen Antagonistic Drugs" and include a conflict level field. Population characteristic incompatibilities cover allergy history tags, pregnancy tags, and chronic disease tags and include trigger condition fields. Dose-related toxicity constraint fragments include dose range identifiers and risk level fields, along with a traceability source field. The knowledge module performs integrity checks, duplicate entry merging, and rule index generation on the drug incompatibilities rule base. The rule index establishes a double-key index based on drug identifier and incompatibility type and generates an index version number. The index version number and the rule base version number are jointly written into the optimization configuration package for subsequent steps to call a consistent version of the rule set when generating a set of legal actions. Synchronized with the incompatibilities rule base, S100 loads the policy network and value network by the evaluation module and performs model file verification and version registration. The policy network is a policy estimation model of Deep Reinforcement Learning (DRL). The model structure includes a state encoding unit, an action encoding unit, a fusion unit, and a probability normalization unit. The state encoding unit receives the disease target and patient characteristics and receives the prescription context placeholder field. The action encoding unit encodes the type of add, remove, and replace actions and embeds the target drug identifier. The fusion unit outputs the action score vector, and the probability normalization unit outputs the action prior probability. The value network is a state value evaluation model of deep reinforcement learning. The model structure includes a state encoding unit, a value item output unit, and a confidence item output unit. The state encoding unit shares the field caliber with the state encoding unit of the policy network and uses the same normalization configuration. The value item output unit outputs the numerical representation of the state value, and the confidence item output unit outputs the confidence representation of the state value. S100 establishes a version record structure for the policy network and the value network. The version record structure includes the model version number, training dataset identifier, training round identifier, and model check code. The version record structure is associated with the session identifier and written into the optimization configuration package. The model check code is generated from the model file content and a consistency comparison is completed during the loading stage, so that the subsequent search tree statistics package based on the optimization configuration package forms an auditable version link.

[0045] After completing field extraction, rule loading, and model loading, S100 encapsulates the disease objective, patient characteristics, number of stages, search budget, rule base version number and index version number of the incompatibility rule base, model version number and model checksum of the policy network, model version number and model checksum of the value network, session identifier, and timestamp to generate an optimized configuration package. This optimized configuration package is then written to the configuration storage area and a configuration reference identifier is registered. The configuration reference identifier and the optimized configuration package serve as input to S200, which uses them to construct stage states and generate stage state packages. These stage state packages are then used as state inputs for Monte Carlo Tree Search (MCTS) in S300 for selection and expansion. In summary, this step achieves the following technical effect: by uniformly loading, verifying, and versioning the disease objective, patient characteristics, number of stages, search budget, incompatibility rule base, policy network, and value network, a consistent optimized configuration package is formed across all steps, providing a traceable configuration basis for subsequent stage state construction and search processes.

[0046] S200: Based on the optimized configuration package, construct the stage status in stages. The stage status includes drug set, patient characteristics, disease goal and stage identifier. When the stage number is greater than the first stage, write the formula code of the previous stage into the stage status to generate the stage status package.

[0047] In one embodiment of the present invention, S200 is triggered by the search module after receiving the optimized configuration package output by S100. The optimized configuration package, as the input of this step, includes disease target, patient characteristics, stage number, search budget, version registration information of the incompatibility rule base, version registration information of the policy network, version registration information of the value network, session identifier, and configuration reference identifier. Specifically, the search module first performs field integrity checks and caliber consistency checks on the optimized configuration package. The field integrity check covers the disease target field, patient characteristic field, stage number field, and search budget field. The caliber consistency check covers the consistency of rule base version registration information and model version registration information under the same session identifier. When a field is missing, the search module writes a missing field marker into the optimized configuration package and records the missing field. When a caliber conflict is detected, the search module writes a conflict record and registers a conflict resolution rule identifier, thereby ensuring that the subsequent stage state construction and search tree statistical records have a traceable configuration source. The disease objective in this step is defined as a structured objective description, including diagnostic codes, syndrome labels, chief symptom labels, objective efficacy labels, and prohibited efficacy labels; the patient characteristics in this step are defined as a set of structured individual characteristics, including constitution labels, allergy history labels, complication labels, previous medication labels, and test summary fields; the number of stages is a parameter for the number of treatment stages and is bound to the stage identifier generation rule; the search budget is a parameter for search resource constraints and is associated with and archived with the iteration scheduling records of subsequent Monte Carlo tree searches.

[0048] After input validation, S200 enters the processing chain for constructing stage states by stage. Specifically, the search module generates a stage number sequence based on the stage number and generates a stage identifier for each stage number according to a preset stage identifier generation rule. The stage identifier consists of a stage name field and a stage number field. The stage name field is obtained by mapping from the stage template library and is linked to the syndrome label in the disease target for correction. The stage template library is maintained within the knowledge module, registered according to the template version number, and bound to the session identifier. The search module then constructs the stage state, which is a unified data structure instance containing four core fields: drug set, patient characteristics, disease target, and stage identifier. The drug set is a formulation context field expressed using a drug identifier sequence. The patient characteristics directly reference the patient characteristic field in the optimized configuration package. The disease target directly references the disease target field in the optimized configuration package. The stage identifier references the stage identifier field output by the stage identifier generation rule. For the initial value of the drug set, the search module writes an empty set and simultaneously writes the drug set code. The drug set code uses a normalized code of the deduplicated drug identifier sequence and is written into the check code field. This empty set and the drug set code constitute one of the minimum parameter sets for this step, which will be directly called in the generation of the legal action set and the construction of the state-action code vector in S300. As an optional extended function, the search module also writes a dose placeholder field and a dosage form placeholder field in the stage state, which are used to complete the field backfilling when the dosage ratio is adjusted in S500. However, the dose placeholder field and the dosage form placeholder field do not participate in the main path of stage state consistency verification output in this step.

[0049] When the stage number is greater than the first stage, S200 also writes the previous stage formula code into the stage state. The previous stage formula code comes from the stage formula package output by S400 in the previous stage under the same session identifier. Specifically, the search module pulls the stage formula package when the stage advancement trigger condition is met. The stage advancement trigger condition is composed of a stage formula package writing completion flag and a stage number increment flag, and the trigger timestamp and trigger source are registered in the scheduling log. The search module extracts the drug set code of the previous stage formula and the stage number code of the previous stage from the stage formula package. At the same time, it retrieves the efficacy embedding code from the efficacy embedding library by drug identifier. The efficacy embedding library is maintained by the knowledge module and registered by library version number. The efficacy embedding code and drug identifier have a one-to-one mapping relationship, and hit records and missing records are written during retrieval. When there is a missing record in the efficacy embedding code, the search module writes a missing mark and adds the corresponding drug identifier to the completion queue. At the same time, the missing mark is written into the previous stage formula code, so that the subsequent state value assessment link of S300 has an auditable input caliber. Subsequently, the search module assembles the drug set code, efficacy embedding code, and stage number code of the previous stage formula into the previous stage formula code, and writes the previous stage formula code into the stage state of the current stage to realize cross-stage formula context transfer.

[0050] After completing the construction of all stage states, S200 encapsulates and generates a stage state package and writes a stage state package reference identifier. The stage state package includes a session identifier, configuration reference identifier, stage sequence number, stage identifier, drug set, drug set code, patient characteristics, disease target, preceding stage formula code, rule base version registration information, policy network version registration information, value network version registration information, and a stage state check code. The stage state check code is generated from the above fields and used for subsequent input consistency verification. The stage state package serves as the input payload for S300, which performs Monte Carlo tree search and generates a search tree statistics package based on it. The rule base version registration information and model version registration information in the stage state package are read and invoked during S300's valid action set verification, action prior probability generation, and state value generation processes. In summary, this step achieves the following technical effect: by solidifying the fields of the optimized configuration package, generating stage identifiers, assembling stage states, and writing preceding stage formula codes, a structurally consistent stage state package is formed, and cross-stage context transfer is completed, enabling subsequent search links to obtain traceable state input.

[0051] S300: Perform Monte Carlo tree search based on the stage state package. The policy network outputs the prior probability of the action and the value network outputs the state value. Calculate the upper confidence bound tree index selection based on the state-action value, number of visits, and prior probability of the action. Generate a legal action set expansion by verifying the addition, removal, and replacement of actions based on the matching taboo rule base, and generate a search tree statistics package.

[0052] In one embodiment of the present invention, S300 is triggered by the search module after receiving the stage status packet output by S200. The stage status packet serves as the input payload for this step and includes a session identifier, stage sequence number, stage identifier, drug set, drug set code, patient characteristics, disease target, previous stage prescription code, version registration information of the incompatibility rule base, version registration information of the strategy network, and version registration information of the value network. Specifically, the search module first performs input caliber verification on the stage status packet. The verification includes checking the stage status check code, checking the consistency of the rule base version registration information, and checking the consistency of the model version registration information. The verification conclusion is written to the search scheduling log. When an anomaly is found during the verification, the search module writes an anomaly flag and registers the anomaly type, anomaly field, and anomaly source. At the same time, the anomaly flag is transmitted along with the stage status packet to the subsequent search tree statistics packet for subsequent steps to trace and locate. Subsequently, the search module parses the stage state packet into the root node state. The root node state corresponds to the stage state of the current stage. The root node registers the node identifier, parent-child relationship placeholder, initial value of access count, initial value of state-action value and candidate action cache in the memory search tree, and binds the stage sequence number and stage identifier to the root node metadata to form a search session instance associated with the session identifier.

[0053] In the candidate action generation chain, the search module constructs a candidate action set based on the drug set in the stage state package. Specifically, the search module extracts a list of candidate drugs from the drug database, which is maintained by the knowledge module and registered according to the database version number. The extraction of the candidate drug list is based on efficacy matching and screening according to the target efficacy tag and prohibited efficacy tag in the disease objective, and combined with the patient characteristics of allergy history tag, complication tag and previous medication tag for population filtering and duplicate medication filtering. The filtered candidate drug list is written to the candidate action cache and the screening log is recorded. Subsequently, the search module generates a candidate action set based on the drug set and the drug candidate list. The candidate action set includes three types of action structures: add actions, remove actions, and replace actions. An add action consists of a target drug identifier, an action type identifier, and an action sequence number identifier; a remove action consists of a drug identifier to be removed, an action type identifier, and an action sequence number identifier; and a replace action consists of a drug identifier to be removed, a target drug identifier, an action type identifier, and an action sequence number identifier. During the generation of the candidate action set, deduplication and mutual exclusion constraint checks are performed. The mutual exclusion constraint check covers duplicate additions of the same target drug identifier, removals and replacements that do not exist in the drug set, and replacements where the target drug identifier and the drug identifier to be removed are the same. The check results are written to the candidate action generation record. This candidate action set, along with the drug set, patient characteristics, and disease target, constitutes the minimum parameter set for the core improvement of this step. Subsequent valid action set verification, action prior probability generation, and upper confidence bound tree indicator selection all directly reference this minimum parameter set.

[0054] In the legal action set generation chain, the search module verifies each candidate action set according to the drug incompatibility rule base and generates a legal action set. Specifically, the search module performs action action processing on the drug set for each candidate action, resulting in a candidate formulation. The candidate formulation includes a candidate drug set and a candidate drug set code. Subsequently, incompatibility conflict detection is performed on the candidate formulation. Incompatibility conflict detection is driven by the rule index and is segmented by incompatibility type. The search content includes drug pair incompatibilities within the candidate drug set, drug and population characteristic incompatibilities between the candidate drug set and patient characteristics, and drug and syndrome incompatibilities between the candidate drug set and disease targets. The matched entries are written to the conflict evidence record. When the incompatibility conflict detection conclusion is negative, the search module writes the corresponding candidate action into the legal action set and simultaneously writes it into the legal action set record field. When the incompatibility conflict detection conclusion is positive, the search module writes a rejection flag for the candidate action and registers the rejection reason code, the matched rule entry identifier, and the conflict level field. The rejection flag is entered into the rejection sub-area of ​​the candidate action cache for subsequent auditing and review. After the set of legal actions is generated, the search module generates a set verification code for the set of legal actions and registers the rule base version registration information, so that the set of legal actions and the stage status package are consistent at the version level.

[0055] In the action prior probability generation chain, the evaluation module invokes the policy network to perform inference on the stage state package and the set of legal actions, outputting the action prior probability and writing it back to the search tree node. Specifically, the evaluation module constructs a state encoding vector for the stage state. The state encoding vector is formed by concatenating the graph embedding of the drug set, the patient feature embedding, the disease target embedding, and the stage identifier embedding. The graph embedding is obtained by index mapping of the drug interaction graph in the graph embedding library, the patient feature embedding and the disease target embedding are obtained by mapping from the dictionary embedding table, and the stage identifier embedding is obtained by mapping from the stage template library. The evaluation module also constructs an action encoding vector for the set of legal actions. The action encoding vector includes the action type encoding and the target drug embedding. The target drug embedding is obtained by searching the efficacy embedding library by drug identifier, and the search hit record is written to the inference log. Subsequently, the evaluation module inputs the state encoding vector and the action encoding vector into the fusion unit of the policy network to calculate the action score vector, and generates the action prior probability through the probability normalization unit. The action prior probability is written back to the prior mapping table of the root node according to the action sequence number, and is also written into the action prior probability record field and bound to the version registration information of the policy network. If the policy network inference returns an anomaly flag, the evaluation module writes the inference anomaly record and sets the prior probability of the corresponding action to a missing test flag. The missing test flag is then entered into the anomaly field area of ​​the search tree statistics package.

[0056] In the state value generation chain, the evaluation module calls the value network to perform inference on the stage state package, outputs the state value, and writes it back to the search tree node. Specifically, the evaluation module constructs a state encoding vector for the stage state and inputs the state encoding vector into the value item output unit and confidence item output unit of the value network to generate value items and confidence items. Then, the value item and confidence item are multiplied to obtain the state value, which is written to the state value field of the root node and bound to the version registration information of the value network. After the state value is written, the search module completes the initial registration of the root node. The initial value of the root node's access count, the initial value of the state-action value, the prior probability of the action, and the state value together constitute the minimum parameter set for calculating the upper confidence bound tree index. This minimum parameter set is continuously updated and written to the search tree statistics package during subsequent selection and expansion processes.

[0057] In the selection and expansion of links, the search module performs iterative scheduling of Monte Carlo tree search based on the upper confidence bound tree metric. Specifically, the search module reads the search budget and generates an iterative scheduling plan, which includes an upper limit on the number of iterations, an upper limit on node expansion, and an upper limit on the number of simulations. The iterative scheduling plan is then written to the search scheduling log. When iterative scheduling starts, the search module performs path selection from the root node downwards in the existing search tree. For each expanded node, the path selection calculates the upper confidence bound tree metric and selects the child node with the largest metric. The upper confidence bound tree metric is obtained by combining a state-action value item, an access count item, and an action prior probability item. The state-action value item comes from the node's cumulative feedback record, the access count item comes from the node's access count record, and the action prior probability item comes from the inference output of the policy network. In the scenario where the number of accesses is zero, the search module uses a smoothed access count registration method to complete the numerical filling required for metric calculation and writes the smoothing mark into the node metadata. After the path selection reaches the expandable node, the search module selects unexpanded actions from the set of legal actions and performs expansion processing. This expansion processing applies the action to the parent node's drug set, generating a child node's stage state and creating a node identifier, parent-child edge identifier, initial visit count, and initial state-action value for the child node. The evaluation module then repeatedly performs action prior probability generation and state value generation on the child node's stage state, writing the action prior probability and state value back to the child node's field area. Simultaneously, it writes the rejection flag and hit rule entry identifier of the expanded action to the edge attribute record, thus forming a traceable expansion link at the action level. If the current node does not have a set of legal actions or all legal actions have been expanded, the search module registers the node as a terminated expansion node and writes the termination reason field. The termination reason field is then entered into the node state record area of ​​the search tree statistics package.

[0058] When the search budget constraint or the node expansion limit is reached, S300 ends the tree construction of the current stage and encapsulates the search tree structure and statistical information within the stage to generate a search tree statistics package. Specifically, the search tree statistics package includes session identifier, stage number, stage identifier, root node identifier, node list, edge list, access count record, state-action value record, action prior probability record, state value record, legal action set record, candidate action generation record, rejection mark record, conflict evidence record, iteration scheduling log, inference log, anomaly mark field area, rule base version registration information, policy network version registration information, and value network version registration information, and generates a statistics package checksum for the search tree statistics package; the search tree statistics package is written to the statistics storage area and a statistics package reference identifier is registered. The statistics package reference identifier and the search tree statistics package are used as input in S400, and S400 simulates and executes the total reward feedback update according to the greedy rolling strategy of the exploration ratio parameter.

[0059] Figure 3Example diagram of MCTS search tree provided in the embodiments of this application, such as Figure 3 As shown, in each iteration of step S300, the system starts from the root state of the current stage (such as an initial prescription or the prescription base determined in the previous stage) and executes a classic MCTS loop. Specifically, the 'selection' phase starts from the root node and recursively applies the upper confidence bound formula to select child nodes with fewer visits or higher value estimates until a scalable leaf node is reached, such as... Figure 3 The path from the root node to leaf node A is shown in the diagram. The 'expansion' phase involves adding one or more legal child nodes to the leaf node based on the action probabilities output by the policy network (e.g., adding 'honeysuckle' or 'forsythia'). Figure 3 The simulation expands from node A to nodes B and C. The 'simulation' phase begins with the newly expanded nodes and rapidly selects several actions based on a simplified strategy (such as a random or greedy strategy) until a simulation termination state is reached, forming a complete draft of the phase recipe. The 'backpropagation' phase then propagates the total multi-objective reward calculated by the evaluation layer at the end of the simulation back along the simulation path, updating the visit count and total value of all nodes on the path, such as... Figure 3 The 'N' value is updated next to the middle node. After multiple iterations, the search tree gradually grows, and the statistical data of the nodes tends to stabilize. Finally, the system determines the final action of the current stage according to a preset strategy (such as selecting the child node with the most visits, or directly selecting the child node with the highest value estimate under the root node), forming the 'stage recipe package' for that stage. This process is repeated in multiple stages, ultimately achieving collaborative optimization of multi-stage recipes.

[0060] In summary, the technical effects of this step are as follows: by mapping the stage state package to an iteratively updatable search tree node, and combining it with the matching taboo rule base to generate a set of legal actions and the reasoning results of the policy network and value network, a traceable search tree statistical package is formed, supporting the continuous execution of subsequent simulations and backhaul links.

[0061] S400: Based on the search tree statistical package, a greedy rolling strategy with exploration ratio parameters is used to simulate and generate the termination state. The efficacy score, safety score, cost score and synergy score are calculated and combined in a multiplicative manner to generate the total reward. The number of visits and the state-action value are updated and the optimal action sequence of the stage is extracted to generate the stage formula package.

[0062] In one embodiment of the present invention, S400 is triggered by the search module after receiving the search tree statistics packet output by S300. The search tree statistics packet serves as the input payload for this step and includes a session identifier, stage number, stage identifier, root node identifier, node list, edge list, access count record, state-action value record, action prior probability record, state value record, legal action set record, iteration scheduling log, inference log, anomaly flag field, version registration information of the matching taboo rule base, version registration information of the policy network, and version registration information of the value network. Specifically, the search module first performs a consistency check on the search tree statistics packet. The check includes checking the statistics packet checksum, checking the reference relationship between the root node identifier and the node list, and checking the key field alignment between the access count record and the state-action value record. Simultaneously, the check conclusion is written to the return scheduling log. When an anomaly flag is found during the consistency check, the search module writes the anomaly flag to the simulation trajectory record for this step and performs isolation marking on the abnormal node, thereby preventing the abnormal node from entering the terminated state simulation link. Subsequently, the search module reads the exploration ratio parameter from the search tree statistics package. The exploration ratio parameter is a randomly selected ratio parameter of the greedy rolling strategy. The exploration ratio parameter is bound to the session identifier and written into the return scheduling log for reuse in subsequent multiple rounds of simulation.

[0063] The greedy rolling strategy is implemented collaboratively by the search module and the evaluation module. At each simulation start, the search module loads the current simulation state from the node state corresponding to the root node identifier. This simulation state is constructed from the stage state field in the node list and includes the drug set, patient characteristics, disease target, stage identifier, and, if the stage number is greater than the first stage, the previous stage's formula code. The search module reads the legal action set record under the simulation state. If the legal action set record is empty or covered by an isolation marker, the current simulation state is registered as a terminated state and written into the termination reason field. If the legal action set record exists, the search module drives action selection according to the exploration ratio parameter. At a ratio equal to one minus the exploration ratio parameter, the action corresponding to the maximum value is extracted from the action prior probability record as the current step action. At the ratio corresponding to the exploration ratio parameter, the current step action is randomly selected from the legal action set record, and the action selection criteria are written into the simulation trajectory record. The execution of the current step action is completed by the search module. The search module applies the current step action to the drug set to generate an updated drug set and a drug set code. Simultaneously, it assembles the updated drug set and drug set code with patient characteristics, disease goals, and stage identifiers to generate an updated simulation state. After the action is completed, the search module calls the incompatibility rule base to perform incompatibility conflict detection on the updated drug set. If an incompatibility conflict is detected, a conflict flag is written to the simulation trajectory record, triggering a termination state determination. If no conflict is detected, the updated simulation state is written to the simulation trajectory record, and the next action selection is initiated. The termination state determination, in addition to the conflict flag, includes three triggering conditions: the action step length reaches a preset upper limit, several consecutive actions do not change the drug set code, and the legal action set record is empty. These triggering conditions are bound to the session identifier and written to the return scheduling log, ensuring that the simulation process maintains a repeatable stopping mechanism under resource constraints. As an engineering implementation example, when this step is triggered by a certain disease target and patient characteristics at the prescription decision terminal, the exploration ratio parameter, the upper limit of the action step size, and the step number threshold that does not change the drug set code continuously are fixed to the back-up scheduling log along with the session identifier. The termination status determination is strictly executed according to the fixed parameters and an auditable simulation trajectory record is generated.

[0064] After the termination state is generated, the evaluation module calculates the efficacy score, safety score, cost score, and synergy score for the termination state, and the search module generates the total reward by combining them multiplicatively. The efficacy score is calculated by the efficacy prediction model, which receives the disease target and drug set in the termination state and outputs the matching probability. The evaluation module performs a log-odds transformation on the matching probability, then generates a target probability based on the target efficacy label in the disease target and calculates the deviation penalty term between the matching probability and the target probability. The log-odds transformation result and the deviation penalty term are then weighted and fused. The fusion result is multiplied by a confidence weight to obtain the efficacy score. The confidence weight is obtained by mapping the output confidence field of the efficacy prediction model and written into the efficacy score record. The safety score is generated by the safety assessment submodule. This submodule retrieves toxicity risk entries corresponding to the drug set in the termination state from the drug toxicity database, reads population characteristic contraindications related to patient characteristics from the incompatibility rule database, and reads syndrome contraindications related to disease objectives. It then combines these with the contraindication conflict markers in the termination state to generate a safety score record. When the contraindication conflict marker is positive, the safety assessment submodule writes the safety score to zero and registers the hit rule entry identifier and conflict level field. When the contraindication conflict marker is negative, the safety assessment submodule generates a safety score based on the toxicity risk level, individual allergy history label hit rate, and complication label hit rate, and writes it to the safety score record. The cost score is generated by the cost assessment submodule. This submodule retrieves the unit cost entry for each drug in the drug set from the drug price database and summarizes it to obtain the total cost. It then aligns this cost with the budget field in the optimization configuration package. The cost assessment submodule generates cost penalties for any amount exceeding the budget and writes them to the cost score record. Simultaneously, it binds the cost score record to the drug set code and stage identifier and writes it to the simulation trajectory record. The collaborative score is generated by the collaborative evaluation submodule. This submodule reads the completed stage formula packages from the multi-stage formula set and extracts efficacy vectors. These efficacy vectors are obtained by decoding efficacy embedding codes retrieved from an efficacy embedding library based on drug identifiers. The collaborative evaluation submodule extracts the corresponding efficacy vectors from the drug set in the termination state, calculates the cosine similarity with the efficacy vectors of each stage formula in the multi-stage formula set, and calculates the intersection-union ratio (IUR) for the drug sets of the two stage formulas. Based on the cosine similarity, the collaborative evaluation submodule generates a complementarity term, and based on the IUR, it generates a redundancy term. Finally, it subtracts the redundancy term from the complementarity term to generate the collaborative score and writes it to the collaborative score record. The search module reads the efficacy score record, safety score record, cost score record, and collaborative score record, and performs a multiplicative combination to generate a total reward. This total reward is bound to the drug set code in the termination state, stage identifier, and simulation trajectory record and written to the total reward record. If a missing test flag exists in the input of the multiplicative combination, a missing test handling rule is triggered. This rule writes the missing test source field into the anomaly flag field area and registers the missing test flag in the total reward record, ensuring a consistent anomaly handling approach for the feedback update.

[0065] After the total reward is generated, the search module performs a backhaul update on the simulated path. The backhaul update traces from the leaf node corresponding to the terminated state along the parent-child relationship to the root node, incrementally updating the state-action value record and access count record for each parent-child edge on the path. Specifically, the search module increments the access count and writes it to the access count record. Simultaneously, it calculates the difference between the total reward and the original state-action value, divides this difference by the updated access count to obtain the update amount, and adds this update amount to the original state-action value to obtain the updated state-action value, writing it to the state-action value record. When a node on the path has an isolation or anomaly marker, the search module adds a backhaul skip marker to that node and only updates the access count record without updating the state-action value record, writing the backhaul skip marker to the backhaul scheduling log. After the backhaul update is completed, the search module registers the simulation number, leaf node identifier, total reward record reference, access count record change summary, and state-action value record change summary in the backhaul scheduling log, forming a traceable update link bound to the session identifier.

[0066] When the search module determines that the simulation count constraint in the search tree statistics package has been met or the stopping condition in the return scheduling log has been met, the search module extracts the optimal action sequence for the stage from the root node and generates a stage formula package. Specifically, the search module reads the access count record in the child edge set corresponding to the root node and selects the action corresponding to the child edge with the largest access count. If there are ties in the access count, the module reads the state-action value record, performs the ties resolution, and registers the resolution rule identifier. The search module iterates the selected actions layer by layer downward according to the parent-child relationship until it reaches the node where the termination state is determined to be true. The action sequence on the path is assembled into the optimal action sequence for the stage and written into the optimal action sequence record for the stage. At the same time, the drug set corresponding to the termination state and the drug set encoding are extracted into a stage formula and written into the stage formula record. The search module encapsulates the session identifier, stage number, stage identifier, stage optimal action sequence record, stage formula record, efficacy score record, safety score record, cost score record, synergy score record, total reward record, simulation trajectory record summary, access count record summary, state-action value record summary, version registration information of the incompatibility rule base, version registration information of the strategy network, and version registration information of the value network to generate a stage formula package. After generating a stage formula package checksum, it writes it to the stage storage area and registers the stage formula package reference identifier. The stage formula package reference identifier, along with the stage formula package itself, serves as input to S500, allowing S500 to aggregate multi-stage formula sets and perform safety verification and dosage ratio adjustment. The technical effect of this step can be summarized as follows: by simulating the termination state of the search tree statistical package, calculating the score, generating the total reward, and updating it back, a closed-loop statistical update chain is formed. After the stopping condition is met, the stage optimal action sequence is extracted to generate the stage formula package, ensuring that subsequent multi-stage formula set aggregations obtain a consistent stage-level input load.

[0067] S500: Based on the phased formulation package, summarize the multi-stage formulation set, perform safety review and dosage ratio adjustment, and generate a multi-stage formulation set.

[0068] In one embodiment of the present invention, S500 is triggered by the interaction module receiving the stage formula package output by S400. The stage formula package serves as the input payload for this step and includes a session identifier, stage number, stage sequence number, stage identifier, stage formula record, stage optimal action sequence record, drug set, drug set code, efficacy score record, safety score record, cost score record, synergy score record, total reward record, simulation trajectory record summary, feedback scheduling log summary, version registration information of the incompatibility rule base, version registration information of the strategy network, and version registration information of the value network. Specifically, the prescription management subsystem retrieves all stage formula packages from the stage storage area according to the session identifier and performs a matching check on the stage number and the number of packages retrieved. At the same time, it performs a continuity check on the stage sequence number and writes the check results to the summary scheduling log. When a stage formula package is missing, a stage sequence number is duplicated, or a stage identifier conflict is detected, the prescription management subsystem generates a summary anomaly mark and registers the missing stage sequence number, conflicting stage sequence number, and corresponding package reference identifier in the stage formula index record, so that the subsequent review link has a traceable input boundary. The prescription management subsystem then sorts the stage prescription packages according to the stage number and writes the sorted package reference identifier, stage identifier, and drug set code into the stage prescription index record. The stage prescription index record serves as the arrangement input within this step, drives the summary assembly process of the multi-stage prescription set, and is written into the output product as a field of the multi-stage prescription set after assembly.

[0069] During the assembly process, the prescription management subsystem performs a standardization mapping process on the drug set in the stage prescription package. This standardization mapping process includes drug identifier alignment, disambiguation of homonyms and heteronyms, alias merging, dosage unit alignment, and dosage granularity alignment. Specifically, the knowledge module provides a drug database containing drug identifiers, standard names, alias sets, dosage form information, unit of measurement sets, common dosage range records, and contraindication association indexes. The prescription management subsystem matches the drug entries in each stage prescription record with the standard names in the drug database, synchronously reads the drug identifiers, and replaces non-standard identifiers in the original drug entries, thereby generating standard drug entry records. For drug entries with ambiguous mappings, the prescription management subsystem reads the target drug embedding reference identifier from the stage optimal action sequence record, performs a similarity comparison with the efficacy embedding code in the efficacy embedding library, selects the candidate standard name with the highest similarity and without conflict triggered by the contraindication association index as the disambiguation result, and writes the disambiguation process into the drug mapping log. For drug entries with missing measurement markers in the dosage field, the prescription management subsystem reads commonly used dosage range records from the drug database and generates initial dosage values ​​by combining patient characteristics such as weight, age, and allergy history. Simultaneously, it writes the missing measurement marker and the source marker of the initial value into the dosage source record. For drug entries with inconsistent dosage units, the prescription management subsystem performs unit conversion according to the unit set in the drug database and writes it into the unit conversion record. After completing the standardization mapping, the prescription management subsystem writes the standard drug entry records for each stage into the multi-stage formulation draft record according to the stage number. The multi-stage formulation draft record includes a session identifier, stage number, stage identifier sequence, drug set for each stage, drug set code for each stage, dosage field for each stage, and version registration information summary field, and is written into the input locations for subsequent safety review and dosage ratio adjustment.

[0070] Safety review is completed collaboratively by the safety assessment submodule and the knowledge module. The safety assessment submodule reads the drug sets, patient characteristics, disease objectives, and stage identifier sequences for each stage from the multi-stage formulation draft record, and calls the incompatibility rule base to perform incompatibility conflict detection on drug combinations within each stage, while simultaneously performing cross-stage incompatibility conflict detection on drug combinations between adjacent stages. Specifically, the incompatibility conflict detection adopts a three-layer rule chain: the first layer performs dual retrieval on the eighteen incompatibilities and nineteen antagonisms related entries; the second layer matches the allergy history label, pregnancy label, and liver and kidney function label in the patient characteristics with the individual incompatibility index in the drug database; and the third layer matches the syndrome incompatibility index corresponding to the stage identifier. The safety assessment submodule writes the hit entry identifier, hit drug pair, hit stage sequence number, and conflict level field of each layer into the safety review record, and writes the conflict level field into the stage formulation index record, so that subsequent dosage ratio adjustments have a conflict gating basis. For conflict level fields reaching the blocking level, the safety assessment submodule registers a review failure flag in the safety review record and writes the review failure flag back to the multi-stage formulation draft record. For conflict level fields at the correctable level, the safety assessment submodule generates a correction suggestion record. The correction suggestion record includes the drug identifier, suggested adjustment direction flag, suggested adjustment range reference flag, and suggested adjustment reason entry flag, and writes the correction suggestion record to the input position of the dosage ratio adjustment. To support the implementation of actual engineering scenarios, when the prescription management subsystem executes this step within the prescription review workflow of the hospital information system, patient characteristics are synchronously pulled from the electronic medical record interface and written to the patient characteristic record, and disease targets are extracted from the outpatient diagnosis record and written to the disease target record. The safety review record is bound to the session identifier and written to the prescription review log library. The prescription review log library provides read-only queries to the prescription management interface, thereby enabling the review entries of the same session to have an audit traceability path.

[0071] Dosage ratio adjustment is performed by the dosage assessment submodule. This submodule reads the multi-stage formulation draft record, the safety review record, and the correction suggestion record, and performs constraint alignment, ratio reshaping, and inter-stage dosage connection processing on the dosage fields of drug entries at each stage. Specifically, the dosage assessment submodule reads the commonly used dosage range record and dosage form information for each drug from the drug database, and performs population adaptation mapping on the commonly used dosage range record by combining the patient characteristics such as weight and age tags, generating a population dosage range record. The dosage assessment submodule performs boundary verification between the dosage fields in the stage formulation record and the population dosage range record, writing an out-of-bounds flag for dosage fields exceeding the boundary and generating a boundary correction amount record. The dosage assessment submodule then reads the optimal action sequence record for the stage and the total reward record, generating a contribution weight record at the drug item level. This contribution weight record is obtained by mapping the order and frequency of actions involving the drug in the action sequence, the inter-stage redundancy item, and the complementarity item of the collaborative scoring record, and is written into the proportional adjustment input. In the proportional reshaping stage, the dosage assessment submodule performs proportional scaling on the drug dosage fields within the same stage according to the contribution weight record. Simultaneously, it performs priority correction on drug items with correction suggestion records, and then performs boundary pullback on drug items with out-of-bounds flags, thereby generating a stage dosage ratio record. For inter-stage dosage connection processing, the dosage assessment submodule reads the adjacent stage identifier and performs transition constraint verification on the dosage fields of drugs sharing the same identifier between adjacent stages. The transition constraint verification matches the dosage change magnitude with the change suggestion index in the drug database. If a match is abnormal, a connection anomaly flag is written, and a connection correction record is generated. The connection correction record includes the adjacent stage sequence number, shared drug identifier, correction direction flag, and correction basis item identifier. The entire adjustment process is written into the dose ratio adjustment record, which includes the original dose field, the adjusted dose field, unit conversion record reference, boundary correction record reference, contribution weight record reference, connection correction record reference, and anomaly marker field area, thus enabling the dose ratio adjustment to have a complete input source, processing path, and output destination.

[0072] After safety review and dosage adjustment are completed, the prescription management subsystem performs final encapsulation on the multi-stage prescription draft record, generating a multi-stage prescription set and writing it into the multi-stage prescription set storage area. Specifically, the prescription management subsystem assembles the session identifier, stage number, stage identifier sequence, drug set for each stage, drug set code for each stage, dosage field for each stage, the safety review record, the dosage adjustment record, stage prescription index record, drug mapping log, summary scheduling log, version registration information of the incompatibility rule base, version registration information of the strategy network, and version registration information of the value network into a multi-stage prescription set, generates a checksum for the multi-stage prescription set, writes it into the multi-stage prescription set storage area, and registers a multi-stage prescription set reference identifier in the prescription management interface. After the multi-stage formula set is written, the interaction module reads the multi-stage formula set reference identifier and loads the drug entries and dosage fields of each stage, and outputs the staged prescription display data to the prescription review terminal; the pharmacy system reads the dosage field and drug identifier in the multi-stage formula set through the prescription management interface to complete the dispensing list generation; the audit system reads the safety review records and dosage ratio adjustment records through the prescription review log library to complete the full-link review and traceability.

[0073] One of the significant benefits of the method described in this invention is its ability to generate multi-stage formulations with clearly defined efficacy targets and synergistic complementarity between stages, rather than just a static, single prescription. Figure 4 The three-stage formulation efficacy vector radar chart provided in the embodiments of this application, such as Figure 4 As shown, the three-stage formula for a specific chronic disease optimized by this method exhibits a clear strategic progression in its efficacy characteristics. In the 'acute phase,' under the strong constraints of efficacy and safety rewards, the optimization algorithm tends to generate formulas primarily focused on eliminating pathogenic factors, which is reflected in peak values ​​on the radar chart for dimensions such as 'relieving exterior symptoms' and 'clearing heat.' Entering the 'stable phase,' the algorithm considers the balance between cost and long-term efficacy more in the reward function. The generated formulas, while continuing to clear residual pathogenic factors, begin to enhance the effects of 'tonifying qi' and 'nourishing yin.' In the 'consolidation phase,' the algorithm focuses more on strengthening the overall treatment and preventing relapse through synergistic rewards. Therefore, the formulas perform optimally in dimensions such as 'tonifying qi,' 'activating blood circulation,' and 'nourishing yin,' improving constitution and consolidating the root cause. The radar chart vividly demonstrates that this method, through Monte Carlo tree search's planning ability for long-sequence decisions and the differentiated guidance of different stage-focused objectives in the multi-objective reward function, can automatically optimize dynamic treatment plans that conform to the TCM concepts of "staged treatment" and "treatment based on symptoms," which is difficult to achieve with traditional single-point optimization or simple enumeration methods.

Claims

1. A collaborative optimization method for multiple traditional Chinese medicine formulas based on Monte Carlo tree search and deep reinforcement learning, characterized in that, include: S100: Obtain disease objectives, patient characteristics, number of stages, search budget, incompatibilities rule base, strategy network and value network, and generate an optimized configuration package; S200: Based on the optimized configuration package, construct the stage status in stages. The stage status includes drug set, patient characteristics, disease goal and stage identifier. When the stage number is greater than the first stage, write the formula code of the previous stage into the stage status to generate the stage status package. S300: Monte Carlo tree search is performed based on the stage state package. The policy network outputs the prior probability of the action, and the value network outputs the state value. The confidence bound tree index is selected based on the state-action value, number of visits, and prior probability of the action; the addition, removal, and replacement actions are verified based on the matching taboo rule base to generate an expansion of the legal action set and generate a search tree statistics package. The process of the policy network outputting the action prior probability includes: constructing a state-action encoding vector for the stage state and candidate actions, wherein the state-action encoding vector includes the graph embedding of the current formula, the action type encoding and the target drug embedding, and performing normalized probability mapping on the state-action encoding vector to generate the action prior probability; The process of the value network outputting state value includes: constructing a state encoding vector for the stage state, and generating a value item and a confidence item based on the state encoding vector, wherein the state value is obtained by multiplying the value item and the confidence item; The set of legal actions is obtained by verifying the set of candidate actions through a rule base for drug incompatibilities. The verification includes: applying the candidate actions to the set of drugs to generate candidate formulations, performing incompatibilities and conflicts detection on the candidate formulations, and writing the candidate actions with negative incompatibilities and conflicts detection results into the set of legal actions. S400: Based on the search tree statistical package, a greedy rolling strategy with exploration ratio parameters is used to simulate and generate the termination state. The efficacy score, safety score, cost score and synergy score are calculated and combined in a multiplicative manner to generate the total reward. The number of visits and the state-action value are updated and the optimal action sequence of the stage is extracted to generate the stage formula package. The greedy rolling strategy includes: selecting the action with the highest prior probability at a ratio minus the exploration ratio parameter, and randomly selecting an action at the ratio of the exploration ratio parameter; S500: Based on the phased formulation package, summarize the multi-stage formulation set, perform safety review and dosage ratio adjustment, and generate a multi-stage formulation set.

2. The method according to claim 1, characterized in that, The preceding stage formulation code includes the drug set code, efficacy embedding code, and stage sequence code of the preceding stage formulation. The efficacy embedding code is obtained by searching the efficacy embedding library according to the drug identifier.

3. The method according to claim 1, characterized in that, The feedback update includes: incrementing the access count by one, and updating the status-action value by dividing the difference between the total reward and the original status-action value by the updated access count.

4. The method according to claim 1, characterized in that, The efficacy score is generated by the matching probability output of the efficacy prediction model for the disease target and the drug set. The calculation of the efficacy score includes: performing a log-odds transformation on the matching probability, calculating the deviation penalty term between the matching probability and the target probability, performing a weighted fusion of the log-odds transformation result and the deviation penalty term, and multiplying the fusion result by the confidence weight to obtain the efficacy score.

5. The method according to claim 1, characterized in that, The calculation of the collaborative score includes: extracting efficacy vectors from two-stage formulations within a multi-stage formulation set and calculating cosine similarity; calculating the intersection-union ratio (IUR) of the drug sets of the two-stage formulations; calculating complementarity and redundancy terms based on cosine similarity and IUR; and generating a collaborative score by subtracting redundancy terms from complementarity terms.

6. The method according to claim 1, characterized in that, The policy network and the value network are obtained through self-play training. The self-play training includes: performing Monte Carlo tree search on the training samples to obtain a training dataset consisting of the state, access number normalization policy, and total reward; calculating the policy loss, value loss, and entropy regularization loss based on the training dataset, and performing a weighted summation of the three types of losses to obtain the training loss; and updating the policy network parameters and value network parameters according to the training loss.

Images