Decision-making method and device based on offline reinforcement learning, equipment and medium

Abstract

The invention relates to the technical field of intelligent decision making, can be applied to business scenes such as financial science and technology and medical health, and discloses a decision making method, device, equipment and medium based on off-line reinforcement learning. The method comprises the steps that a basic model is obtained, adjustment is conducted based on domain knowledge, an off-line data set is constructed, and a decision making strategy is trained through off-line reinforcement learning; and integrating the trained decision strategy to an intelligent agent, generating a decision action and decision explanation information corresponding to the current state information in a decision environment, performing compliance examination on the decision action and the decision explanation information, and executing the decision action when the compliance examination is passed to obtain a decision result. According to the method, the domain knowledge is introduced to adjust the basic model, the decision strategy is optimized by using offline reinforcement learning, and the decision explanation information is combined to carry out compliance examination, so that the agent can complete decision optimization without online exploration, and the transparency and credibility of the decision are improved.

Description

Technical Field

[0001] This invention relates to the field of intelligent decision-making technology, and in particular to a decision-making method, apparatus, device, and medium based on offline reinforcement learning. Background Technology

[0002] In the fintech sector, with the widespread adoption of large-scale pre-trained models and machine learning technologies, more and more institutions are attempting to apply these models to risk control, product pricing, risk assessment, and claims processing. However, existing technologies still suffer from structural deficiencies in practical implementation. Existing systems typically use pre-trained models as static tools, performing only fixed tasks such as text generation, information extraction, or rule-based decision-making, lacking adaptive learning capabilities for dynamic environments. When models are used for decision-making tasks, they often rely on supervised offline training, failing to effectively incorporate the latest business feedback for strategy optimization. Furthermore, the high-risk, high-cost, and heavily regulated nature of financial and insurance businesses makes online trial-and-error extremely costly, hindering model updates in real-world business environments and limiting the speed of intelligent decision-making evolution. Existing models often exhibit insufficient adaptability to changing policy requirements, market fluctuations, and user behavior patterns, making it difficult to support long-term, stable strategy optimization needs. Simultaneously, when executing critical business decisions, model outputs lack necessary interpretability, limiting their deep application in financial scenarios with stringent compliance reviews.

[0003] In the healthcare sector, intelligent decision support systems face similar challenges. Existing technologies typically provide diagnostic suggestions, risk alerts, or treatment assistance to healthcare professionals based on fixed rules or supervised learning models. However, these systems struggle to continuously improve their decision-making capabilities based on clinical feedback. The healthcare environment is highly dynamic, with rapid changes in patient conditions, complex treatment processes, and constantly evolving medical knowledge, making static models often inadequate for adapting to new business needs. Furthermore, decision-making activities in healthcare settings are subject to strict regulatory guidelines and ethical constraints, but existing models often generate treatment recommendations with insufficient interpretability, failing to meet the transparency and traceability requirements of clinical applications. Due to the lack of stable learning mechanisms based on real-world business feedback, these decision-making models struggle to achieve continuous optimization and provide stable and reliable recommendations in clinical decision support. Summary of the Invention

[0004] The main objective of this invention is to provide a decision-making method, apparatus, device, and storage medium based on offline reinforcement learning, aiming to solve the technical problems of existing technologies that cannot build a basic model into an intelligent agent with continuous evolution capabilities, and cannot use historical decision data to complete interpretable and auditable offline decision optimization without online trial and error, thus making it difficult to adapt to dynamic business environments.

[0005] To achieve the above objectives, this invention provides a decision-making method based on offline reinforcement learning, comprising: Obtain a base model, adjust the base model based on domain knowledge to obtain an adjusted base model, encapsulate the adjusted base model into an agent, and define a state space, action space and reward function for the agent; An offline dataset containing historical state information, historical action information, historical reward information, and historical next state information is constructed based on historical decision data. Based on the offline dataset, an offline reinforcement learning module is used to train the decision strategy. The trained decision-making strategy is combined with the agent to form a deployment agent; In the decision-making environment, the deployed intelligent agent receives and processes the current state information, and outputs decision actions and corresponding decision explanation information. The decision-making actions and the decision explanation information shall be subject to compliance review; When the compliance review is passed, the decision-making action is executed and the decision result is obtained.

[0006] Furthermore, to achieve the above objectives, the present invention provides a decision-making device based on offline reinforcement learning, comprising: The model building module is used to obtain a basic model, adjust the basic model based on domain knowledge to obtain an adjusted basic model, encapsulate the adjusted basic model into an agent, and define a state space, action space and reward function for the agent. The offline reinforcement learning module is used to construct an offline dataset containing historical state information, historical action information, historical reward information, and historical next state information based on historical decision data. Based on the offline dataset, the offline reinforcement learning module is used to train the decision strategy. The agent integration module is used to combine the trained decision-making strategy with the agent to form a deployment agent; The decision generation module is used to receive and process current state information through the deployed intelligent agent in the decision-making environment, and output decision actions and corresponding decision explanation information; The compliance review module is used to conduct compliance reviews of the decision-making actions and the decision explanation information. The decision execution module is used to execute the decision action and obtain the decision result when the compliance review is passed.

[0007] Furthermore, to achieve the above objectives, the present invention also provides a computer device, the computer device including a memory, a processor, and a decision-making program based on offline reinforcement learning stored in the memory and executable on the processor, wherein when the decision-making program based on offline reinforcement learning is executed by the processor, it implements the steps of the decision-making method based on offline reinforcement learning as described above.

[0008] Furthermore, to achieve the above objectives, the present invention also provides a non-volatile computer-readable storage medium storing a decision-making program based on offline reinforcement learning, wherein the decision-making program based on offline reinforcement learning, when executed by a processor, implements the steps of the decision-making method based on offline reinforcement learning as described above.

[0009] Beneficial Effects: This invention relates to the field of intelligent decision-making technology and can be applied to business scenarios such as fintech and healthcare. It discloses a decision-making method, apparatus, device, and medium based on offline reinforcement learning, comprising: acquiring a basic model and adjusting it based on domain knowledge; constructing an offline dataset and training a decision strategy using offline reinforcement learning; combining the trained decision strategy with an intelligent agent to form a deployed intelligent agent; receiving current state information in the decision environment and generating decision actions and corresponding explanatory information; conducting compliance reviews of the decision actions and explanations; and executing the decision action to obtain the decision result when the compliance review is passed. This invention, by adjusting the basic model into an intelligent agent capable of executing decisions, combining it with a decision strategy trained through offline reinforcement learning, and generating and conducting compliance reviews of decision explanations during the inference phase, achieves improved decision quality using historical data without online trial and error. This approach completes the transformation from a static model to a dynamically interpretable intelligent agent, making the decision-making process more transparent, reliable, and meeting compliance requirements in the business environment, thereby improving decision optimization capabilities and application security. Attached Figure Description

[0010] The present invention will be further described below with reference to the accompanying drawings and embodiments. In the accompanying drawings: Figure 1 This is a schematic diagram of an application environment for a decision-making method based on offline reinforcement learning in one embodiment of the present invention; Figure 2 This is a flowchart illustrating an embodiment of the decision-making method based on offline reinforcement learning according to the present invention. Figure 3 This is a schematic diagram of the functional modules of a preferred embodiment of the decision-making device based on offline reinforcement learning of the present invention; Figure 4 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention; Figure 5 This is another structural schematic diagram of a computer device according to one embodiment of the present invention. Detailed Implementation

[0011] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0012] The decision-making method based on offline reinforcement learning provided in this invention can be applied to, for example... Figure 1 In this application environment, the client communicates with the server via a network. The server can obtain a basic model from the client and adjust it based on domain knowledge, construct an offline dataset, and train a decision-making strategy using offline reinforcement learning. The trained decision-making strategy is then combined with an intelligent agent to form a deployed agent. In the decision-making environment, the agent receives current state information and generates decision actions and corresponding explanations. Compliance reviews are conducted on the decision actions and explanations, and the decision action is executed to obtain the decision result when the compliance review is passed. This invention achieves improved decision quality using historical data without online trial and error by adjusting the basic model to an intelligent agent capable of making decisions, combining it with decision-making strategies trained through offline reinforcement learning, and generating and reviewing decision explanations during the inference phase. This method completes the transformation from a static model to a dynamic, interpretable intelligent agent, making the decision-making process more transparent, reliable, and meeting compliance requirements in the business environment, thereby improving decision optimization capabilities and application security. The client can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented using a standalone server or a server cluster consisting of multiple servers. The invention is described in detail below through specific embodiments.

[0013] Please see Figure 2 , Figure 2 This is a flowchart illustrating an embodiment of the decision-making method based on offline reinforcement learning provided by the present invention. It should be noted that although the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0014] like Figure 2 As shown, the decision-making method based on offline reinforcement learning proposed in this invention includes the following steps: S10, Obtain the basic model, adjust the basic model based on domain knowledge to obtain the adjusted basic model, encapsulate the adjusted basic model into an agent, and define the state space, action space and reward function for the agent; In this embodiment, after acquiring the base model, to ensure the model's ability to adapt to the domain environment, it needs to be adjusted using domain knowledge. The base model typically originates from a pre-trained model system, such as a large-scale parameter network built using a large amount of general text or data. These models possess cross-task expressive capabilities but lack discrimination or processing logic specific to particular decision-making scenarios. Therefore, introducing domain knowledge can compensate for the base model's shortcomings in professional knowledge. Sources of domain knowledge include manually compiled expert rules, structured or unstructured professional statements in domain business data, and long-accumulated experience samples. Domain knowledge is organized as data points, enabling the base model to absorb new knowledge structures through parameter updates, making the model more closely aligned with the semantic expressions, information relationship structures, and behavioral logic within the business domain. The adjustment process is implemented through supervised training, inputting domain data into the model to shift the base model's internal parameters towards a direction more suitable for the business context.

[0015] After adjusting the basic model, it needs to be encapsulated into an intelligent agent. Encapsulation means endowing the model with action capabilities, state processing capabilities, and reward feedback capabilities, transforming it from a static model into an entity capable of making decisions. The encapsulation process typically involves embedding the model into a structure with state input interfaces, action generation interfaces, and feedback processing interfaces, enabling the model to interact with the outside world in the decision-making environment. The key to forming an intelligent agent lies in defining its state space. The construction of the state space originates from all observable variables provided by the environment, including both environmental feature vectors and internal state vectors formed by the agent during operation. For example, environmental feature vectors can consist of business indicators, behavioral parameters, and profile variables, while internal state vectors can consist of the model's hidden layer representation or short-term memory structure. By combining information from different sources into a unified structure, the intelligent agent can accurately judge its environment.

[0016] The action space is defined by the set of decisions an agent can execute in a specific scenario. Actions can be represented as discrete choices, such as approving or rejecting in a risk assessment, or as continuous values, such as outputting a value within a range in a continuous control scenario. The definition of the action space affects the types of decisions an agent can make, and therefore needs to be constructed according to the business task structure.

[0017] The reward function measures the quality of an action and forms the basis for an agent's behavioral strategy. It is generated based on performance metrics after an action is performed and can employ a single-metric weighted approach or a combination of multiple performance metrics. The reward value numerically represents the action's effect in the real-world environment, such as decision accuracy, cost impact, and error rate changes. During training, the reward value is received by the agent and used to adjust its strategy, ultimately leading to a behavioral pattern that meets long-term performance requirements.

[0018] This embodiment adjusts the basic model by introducing domain knowledge, and then forms an intelligent agent through state space combination, action space construction and reward function definition. This enables the model to understand the business environment, execute decision-making actions, receive feedback results and use the feedback in subsequent training, realizing the transformation from a static model to an interactive entity, and improving the adaptability, plasticity and decision-making accuracy of the intelligent agent in professional fields.

[0019] S20, construct an offline dataset containing historical state information, historical action information, historical reward information, and historical next state information based on historical decision data, and train the decision strategy using an offline reinforcement learning module based on the offline dataset; In this embodiment, when constructing the system based on historical decision-making data, it is first necessary to extract information related to decision-making behavior from long-term accumulated business operation records. These records contain the environmental performance encountered by the system at different time points, feedback from manually executed actions or system-executed actions, and subsequent changes triggered by these actions. Therefore, they can reflect the complete experience trajectory formed by the decision-making system in actual operation. Historical decision-making data usually comes from log recording systems, audit process systems, or behavior tracking systems, and contains multi-source information structures. By organizing this data in a structured manner, historical state information can be identified. Historical state information is used to describe the environmental conditions in which the system is at a certain moment, including behavioral signals, user characteristics, business variables, etc., which constitute an observable description of the environment.

[0020] Historical action information refers to the actual actions taken at the corresponding moments in historical state information. These can be discrete choices or continuous control variables, serving as a crucial basis for constructing the action space reference range. Historical reward information expresses the performance of historical actions in the business environment at the time, abstractly expressed through single-action gains or long-term gains, and is typically represented numerically in data processing. Historical next-state information characterizes the environment the system enters after an action is executed. This information reflects the impact of the action on the overall system environment and also demonstrates the causal structure of the environment as the action changes. By combining historical state information, historical action information, historical reward information, and historical next-state information, the foundation of the offline dataset is formed.

[0021] The offline dataset is constructed following the principle of time-series consistency, ensuring that each historical record entry represents a continuous correspondence between state, action, reward, and the next state. Once constructed, the offline reinforcement learning module uses this dataset as its training input. The offline reinforcement learning module's structure may include a policy network, a value function network, and a stable training control unit. Its function is to learn behavioral policies using fixed data without online interaction. Since all training data comes from historical decision data, the module needs to avoid policy shift during training by constraining the action distribution in historical trajectories to ensure that policy learning does not deviate from the executable range of real-world business scenarios. During training, the offline reinforcement learning module calculates the correlation between states and actions, as well as estimates of long-term rewards, to gradually form an optimized decision policy, enabling it to maintain stable performance even in new environments.

[0022] This embodiment constructs an offline dataset based on historical decision data and trains the decision-making strategy using an offline reinforcement learning module. This separates the training process from the real business environment, reducing interaction costs and risk exposure. At the same time, long-term historical trajectories are incorporated into the strategy optimization process, enabling the strategy to have a more comprehensive ability to learn behavioral patterns and maintain stable performance in future environments.

[0023] S30, combine the trained decision-making strategy with the intelligent agent to form a deployment intelligent agent; In this embodiment, the decision policy obtained after training belongs to the behavior mapping structure formed by structured learning based on historical decision trajectories during the offline reinforcement learning stage. This mapping structure is used to generate the mapping relationship for execution behavior based on the current state. This mapping relationship includes a parameter matrix, a weight space, and an internal computational path for inference, which together define the output behavior of the policy in different states. To enable the policy to have actual execution capability, it needs to be combined with the agent after domain knowledge adjustment, allowing it to form a linked structure with the agent's internal inference module, environment interface module, and description generation module, thereby achieving executability.

[0024] The integration process comprises two parts: logical integration and structural integration. Logical integration refers to establishing a correspondence between the decision-making strategy and the agent's internal state update structure, input processing structure, and output feedback structure during the reasoning process. This ensures that the input received by the strategy is consistent with the state representation provided by the agent, thereby guaranteeing that the strategy reasoning result has a correct data source. Structural integration involves embedding the strategy reasoning module into the agent framework, making it a callable functional unit within the framework. This includes establishing a strategy entry point in the agent's reasoning pipeline, adding strategy output nodes to the reasoning chain, and using strategy signals at the nodes where the agent executes actions. Through this structural integration approach, the strategy and the agent form a stable operational structure capable of processing input states, triggering reasoning, and generating behavioral outputs.

[0025] The process of forming a deployed agent requires ensuring the consistency of input and output formats across all modules. For example, the state representation should maintain its vectorized structure when entering policy inference, and the policy output and the action interface of the agent's behavior execution module need to be consistent in form to ensure that the inference results can be directly read by the execution unit. Simultaneously, the overall operation of the agent after policy embedding needs to pass consistency checks, including whether policy parameters can be invoked, whether the inference path is coherent, and whether the action output is compatible with the execution module, to ensure that the combined structure can execute stably in a real-world environment. Once combined, the agent possesses the ability to receive states, invoke policy inference, and output behaviors in a real-world environment, becoming an executable deployed agent.

[0026] This embodiment combines the trained decision-making strategy with the intelligent agent, enabling the behavioral mapping capabilities obtained during the training phase to enter the runtime environment. This connects data-driven learning with actual execution, giving the intelligent agent a feasible decision-making capability, reducing strategy deployment costs, and improving the stability and consistency of decision execution in the real environment.

[0027] S40, In the decision-making environment, the deployed intelligent agent receives and processes the current state information, and outputs decision actions and corresponding decision explanation information; In this embodiment, in the environment where the agent is deployed, it needs to be able to receive descriptive information corresponding to the current business context from the external environment. The current state information is a structured representation that jointly expresses the external context, internal variables, and entity attributes. It is transmitted to the agent through the environment interface to form the input conditions for decision-making inference. The current state information may exist as a vector, feature list, temporal encoding, or other inference-compatible representations, used to characterize the feature state of the business object at a certain moment. The receiving process typically requires pre-built parsing logic within the agent to perform format conversion, numerical standardization, field filtering, or temporal alignment of the raw input data, making it directly usable by the policy inference network.

[0028] After receiving the current state information, the deployed agent needs to process it through an internal policy network. The policy network consists of parameter mappings obtained during the training phase and is used to generate decision-making action outputs based on the current state information. Internal processing includes multiple computational paths, such as feature projection, nonlinear transformation, weight calculation, and action preference value calculation. Through this reasoning process, the policy network can deduce behavioral outputs suitable for the current business context from the current state information. Decision-making actions are a type of well-defined behavioral signal, which may be a discrete choice, a continuous variable, or a segmented interval, used to represent the agent's response to the current state and reflect the agent's behavioral tendencies.

[0029] After a decision is made, explanatory information is needed to explain the behavioral structure, making the decision-making process understandable. This explanatory information is generated by the base model module within the deployed agent. This base model module is derived from a structure adjusted for domain knowledge, possessing the ability to process natural language descriptions, generate textual expressions, and parse internal reasoning paths. The explanatory information is typically generated based on the relationship between the current state information and the decision action, potentially relying on weight contributions, feature importance, business rule mappings, or behavioral logic cues, explaining the basis for the decision action in natural language.

[0030] Finally, the intelligent agent is deployed to output decision-making actions and corresponding explanatory information to an external system. The output structure needs to undergo internal formatting to ensure that the action signals and explanatory text are structurally consistent with the interface of the external system. This includes action type encoding, text content formatting, and the establishment of a result transmission mechanism. Through this operational structure, the intelligent agent can form a closed-loop reasoning chain in the decision-making environment, transforming current state information into behavioral actions and explanatory content, thus achieving executable and interpretable behavioral response capabilities.

[0031] This embodiment deploys an intelligent agent in the decision-making environment, enabling it to generate decision actions and natural language explanations based on current state information. This achieves a coherent process of receiving environmental information, reasoning about strategies, and interpreting behaviors, improving the understandability of behavioral outputs, making the action execution process more transparent, and providing a structured basis for subsequent compliance review, risk control, and business verification.

[0032] S50, conduct a compliance review of the decision-making action and the decision explanation information; In this embodiment, the compliance review process for decision-making actions and explanations involves systematically verifying the behavioral outputs and their explanatory content to ensure they comply with industry standards, business regulatory requirements, and internal control standards. Decision-making actions originate from the agent's behavioral choices in the current environment and typically exist in the form of classification labels, numerical variables, or range parameters, representing the intent to execute a particular behavior or business operation. Decision explanations originate from natural language expressions generated by the basic model module, explaining the basis, triggering conditions, or strategy logic for the action. Both types of content require compliance review to ensure that the behavior does not violate regulatory requirements, operational procedures, industry rules, or ethical restrictions.

[0033] During operation, decision actions need to be structurally matched using pre-configured sets of rules, strategies, or text filters. Decision actions are well-structured instructions that can be verified through features such as action encoding, action type, and parameter range. For example, this can be used to determine whether the action exceeds business limits, involves restricted behaviors, or violates existing constraints. Decision explanation information is natural language text; therefore, the text content needs to be parsed and mapped to auditable expression units, analyzed through methods such as sensitive word detection, semantic deviation identification, logical consistency verification, or industry standard comparison.

[0034] Compliance reviews typically consist of three consecutive stages: rule matching, content analysis, and comprehensive assessment. Rule matching identifies whether an action falls within the permissible range or triggers regulatory rules. Content analysis identifies whether explanatory text contains inappropriate expressions, misleading information, or potential biases. Comprehensive assessment integrates the analysis results from the two stages to determine whether the action is permissible to proceed, and may simultaneously generate a review record, providing a basis for subsequent archiving, regulatory tracking, or risk tracing. Ultimately, the review conclusion serves as a signal for whether the action can proceed to the next stage, determining the system's subsequent execution path.

[0035] This embodiment conducts compliance reviews of decision-making actions and explanatory information, ensuring consistency verification and risk filtering of behavioral outputs before they enter the business execution stage. This helps reduce the spread of violations, misleading statements, and inappropriate behaviors, strengthens process controllability, and achieves the goals of controllable behavior, credible explanations, and stable business operations.

[0036] S60, when the compliance review is passed, the decision action is executed and the decision result is obtained.

[0037] In this embodiment, the execution decision action belongs to the operation execution stage after the compliance review is completed in the preceding processing link. "Compliance review passed" indicates that the review result formed in the preceding link meets the execution conditions. This result typically originates from structured verification logic, rule-based judgment processes, or review judgments supported by models. This result serves as a trigger signal to initiate the action execution process. The "decision action" involved in action execution originates from the action instructions output by the deployed intelligent agent when processing the current state information. These instructions can be represented as symbolic operation markers, numerical action parameter sets, or serialized structured action descriptions, and are used to be parsed in the execution unit to drive subsequent processes.

[0038] The execution of an action first requires format parsing of the action command. The parsing module identifies the action's parameter fields and behavior type, then maps the parsed action content to the executable interface of the execution subsystem. The execution subsystem can be a service interface, business processing unit, external action proxy module, or device control module, and typically includes components such as action scheduling, parameter translation, and execution status feedback. The parsed action is assigned to the corresponding execution path by the scheduling component, and parameter translation converts the abstract action expression into an operation language or format that meets the requirements of the execution unit. After receiving the translated action content, the execution unit performs the actual operation, including triggering internal logical operations or calling external systems.

[0039] During execution, feedback information needs to be generated to indicate the progress, status, or intermediate results of the action. This feedback information can be generated by monitoring the operational status of the execution unit, such as monitoring function call return values, device action status, process node result markers, or business operation completion markers. This information is then combined according to a unified structure to form the execution result. After execution, the final decision is determined based on the obtained feedback data. This decision result can be a structured record of operational behavior results, a summarized numerical output, a model-readable format of status update information, or a user-readable description of the final action effect.

[0040] This process forms a complete action implementation chain, including action execution, action parsing, parameter translation, execution status feedback, and decision result synthesis. This enables instructions from the agent to be transformed from abstract action expressions into executable behaviors, and thus obtain the final output.

[0041] In implementation, different action execution mechanisms can be used to support the landing of actions in different scenarios. For example, a service interface-based execution method can be adopted, mapping action instructions to internal service interfaces, translating parameters through a lightweight parameter mapping module, and then having the service logic execute the operation, extracting execution status information from the service return value. Alternatively, an external device control-based execution method can be used, mapping actions to device control protocols, sending action parameters through communication interfaces, obtaining execution progress and end markers through a device status feedback mechanism, and assembling decision results accordingly.

[0042] Actions can also be implemented through a workflow execution engine, transforming action instructions into workflow node calls. The workflow engine's node status recording capabilities enable execution status tracking and result aggregation to adapt to multi-node, multi-step business actions. For execution scenarios requiring high reliability, exception rollback logic can be added. When an action fails, an error flag is recorded and a compensation process is triggered. The compensation module corrects the execution result or re-triggers the action execution.

[0043] When adapting to different business environments, dynamic parameter mapping strategies can be used to adjust the interpretation logic of action parameters, allowing the same action instruction to be mapped to different execution methods in different environments. For example, on resource-constrained devices, actions can be implemented by simplifying execution parameters and reducing communication overhead; in high-performance systems, more granular result output can be obtained by expanding parameter dimensions and adding execution paths. Furthermore, in data processing environments, caching intermediate execution results can reduce latency caused by repeated execution, making action execution more efficient.

[0044] This embodiment ensures the controllability, traceability, and feasibility of the action output process by executing decision-making actions and generating decision results after compliance review is passed. It guarantees the rationality of action input during the review stage, and then forms an executable behavior chain through clear parsing, scheduling, translation, and feedback mechanisms during the execution stage. This achieves a coherent transformation from abstract action description to actual execution output, and obtains decision results that meet the review requirements.

[0045] In one embodiment, step S10 includes: S101, Obtain the pre-trained base model; S102, Construct a domain knowledge corpus containing domain-specific knowledge data points; S103, Based on the domain knowledge corpus, the basic model is supervised and adjusted to obtain the adjusted basic model; S104, Integrate the adjusted basic model into the agent framework as the core reasoning module of the agent; S105, Define a state space for the agent, the state space being a combination of environmental feature vectors and agent internal state vectors; S106, Define an action space for the agent, the action space including a set of discrete or continuous decision actions; S107, Define a reward function for the agent, wherein the reward function determines the reward value based on a weighted average of performance metrics after the decision action is executed.

[0046] In this embodiment, the pre-trained base model is a model instance that has already completed parameter training on large-scale general data. It can be a language model trained on a dataset primarily consisting of natural language text, or a multimodal model containing both structured and textual features. When obtaining the pre-trained base model, a specified version of the weight file and structure definition can be pulled from a model repository, while simultaneously recording model configuration parameters, including the number of network layers, hidden dimensions, number of attention heads, and activation function type. After loading, weight initialization and operator compilation need to be completed in the computing environment so that the base model can be directly called during subsequent training and inference. Through this process, the decision optimization process does not need to build capabilities from scratch, but rather is built upon existing representation and inference capabilities.

[0047] Domain-specific knowledge data points describe knowledge units within a target business domain. These can be a single policy clause, a medical guidance statement, a claims rule, a health management suggestion, a tagged dialogue, or a structured record composed of multiple fields, such as a combination of customer profile fields, behavioral tags, and risk level tags. When constructing a domain knowledge corpus containing these data points, data can be collected from various sources, including historical business documents, business rule bases, dialogue records, and business processing logs. Through cleaning, deduplication, anonymization, and format standardization, a consistent and clearly labeled sample set is obtained. The corpus can be categorized according to task type, such as "risk assessment," "product matching," and "health intervention suggestions," to allow for the selection of different subsets for training during the monitoring and adjustment phase.

[0048] The supervised adjustment process, based on a domain knowledge corpus, performs targeted parameter updates to the base model. Specifically, it constructs input-output pairs, shaping the input into combinations of contextual information, problem descriptions, and structured business features, while shaping the output into desired decision suggestion text, label sequences, action categories, or multi-dimensional scoring vectors. By defining a loss function, the model output is compared to the reference answers carried in the domain knowledge data points, such as using cross-entropy loss, sequence labeling loss, or multi-task joint loss. Backpropagation is then used to update the base model parameters, ensuring the model maintains general expressive power while more closely approximating the expected response of the target domain. The adjusted base model has a parameter distribution more closely aligned with the target business scenario, providing more robust language output and semantic representation when facing domain tasks.

[0049] When the adjusted base model is integrated into the agent framework as the core inference module, an encapsulated structure is needed that includes state access, action output, and interaction with the external environment. The agent framework typically includes components such as a decision-making strategy interface, an environment interaction interface, a memory unit, and a log recording unit. When injecting the adjusted base model into the core inference location, the model's capabilities can be split into two layers: firstly, acting as a state encoder to encode environmental information, historical dialogues, and business features into high-dimensional vector representations; secondly, acting as an explanation generator to generate explanatory text after the decision action is determined. During integration, a unified calling interface needs to be designed so that the decision-making strategy module can request embedded representations or generated content from the core inference module through a unified data structure, and return the results to the decision-making process after completing the inference.

[0050] The state space is used to characterize the range of information that an agent can perceive during decision-making. When the state space is composed of environmental feature vectors and the agent's internal state vectors, it can be uniformly represented through vector concatenation, linear mapping, or multi-head attention fusion. Environmental feature vectors represent external information relevant to the current business scenario. For example, in financial transactions, this includes customer attributes, historical transaction summaries, current product configurations, and external market indicators; in healthcare transactions, it includes user health records, recent behavior records, and real-time monitoring data. Environmental feature vectors can be generated through encoding networks, feature embedding layers, or rule mapping modules, ensuring that each dimension has a clear meaning or can be learned during training.

[0051] An agent's internal state vector represents its internal memory and operational context, such as decision history, previous actions, internal belief states, and unfinished subtask markers. Internal states can be maintained through recursive structures, memory networks, buffer queues, or key-value stores. Before each decision, the latest internal state is jointly input into the policy module along with the environmental feature vector. This combination ensures that the state space contains not only information from a single observation of the environment but also historical information across time and intermediate results from internal inference, enabling the decision-making process to leverage long-term dependencies.

[0052] The action space describes the set of all decision actions an agent can take, which can include discrete and continuous actions. Discrete decision actions represent a finite number of optional operations, such as selecting a risk level, triggering a risk control strategy, or choosing a product recommendation in financial transactions; or sending an intervention suggestion of a certain type, selecting a follow-up plan with a certain frequency, or switching a monitoring mode in healthcare. Continuous decision actions represent parameterized control variables, such as adjusting premium ratios, setting risk thresholds, configuring intervention intensity, or determining intervention intervals. Each element in the action space needs to have a clear business meaning and a definite mapping relationship with the specific operation interface of the execution system. When constructing the action space, discrete actions can be encoded as indexed or one-hot vectors, and continuous actions can be represented as real-number vectors, and prediction can be performed in the policy network using the corresponding output layer form.

[0053] The reward function evaluates the quality of the results after a decision action is executed. It determines the reward value based on a weighted average of performance metrics after the action. Performance metrics can include multiple dimensions; for example, in financial businesses, these include profitability, default rate, payout rate, customer retention, and complaint rate; in healthcare businesses, they include symptom relief, compliance metrics, long-term risk score changes, and intervention completion rate. Performance metrics can be calculated through historical data statistics, real-time system monitoring, or post-hoc evaluation modules. The reward function can be implemented using a weighted summation structure, combining multiple performance metrics with preset or dynamic weights to form a scalar reward used to train the decision strategy. Weights can reflect business preferences; for example, increasing weights related to risk metrics when emphasizing risk control, and increasing weights related to return metrics when emphasizing return. These weights can also be adjusted through automated strategy search. The reward function can also introduce penalty terms, assigning negative rewards to decisions that violate business constraints or cause high costs, guiding the agent to gradually avoid undesirable behaviors during training.

[0054] This embodiment acquires a pre-trained basic model and performs supervised adjustment in conjunction with a domain knowledge corpus, transferring the capabilities of a general model to specific business scenarios, making the agent more aligned with domain requirements in terms of language understanding and semantic expression; by integrating the adjusted basic model into the agent framework as the core reasoning module, and clarifying the structure of the state space, action space, and reward function, the decision-making process has computable state inputs, executable action outputs, and quantifiable feedback signals.

[0055] In one embodiment, step S20 above includes: S201, Obtain historical decision log data from the business system; S202, preprocess the historical decision log data to obtain preprocessed historical data; S203, extract historical state information, historical action information, historical reward information, and historical next state information from the preprocessed historical data; S204, Construct an offline dataset based on the historical state information, historical action information, historical reward information, and historical next state information; S205, Initialize the offline reinforcement learning training environment and load the offline dataset; S206, Configure the network structure and training hyperparameters for the offline reinforcement learning module; S207, The offline reinforcement learning module trains the policy network based on the offline dataset using the conservative Q-learning method, and introduces a behavior cloning regularization term during the training process to obtain the trained decision policy.

[0056] In this embodiment, the business system can be a risk control and transaction system or a product recommendation system of a fintech platform, or a follow-up management system or a health intervention platform in a healthcare scenario. Historical decision log data refers to the collection of raw records automatically recorded during actual business operations, typically including fields such as request time, user identifier, environmental feature snapshot, system-provided decision action, business execution result, and subsequent feedback markers. To facilitate the subsequent construction of training data, it is necessary to ensure the stability of the log structure and the clear definition of fields during collection, and to aggregate logs from multiple time periods into a unified storage through batch retrieval interfaces or offline export tasks.

[0057] The preprocessing stage systematically addresses noisy records, missing fields, and outliers in historical decision log data. The preprocessed historical data emphasizes two types of information: environmental and contextual fields used to construct state representations, and feedback fields used to measure action outcomes. Specifically, deduplication is performed first to remove duplicate records, followed by a field integrity check to filter out records with missing key fields. Outliers are then truncated or recoded; for example, obviously unreasonable amounts, scores, or indicator values ​​are merged to preset upper or lower bounds. For categorical fields, a unified encoding table aligns values ​​from different sources to a unified label. For numerical fields, standardization, interval scaling, or quantile transformations are used to ensure features have a more stable numerical range when input into the training module. The preprocessed historical data retains the information needed for a complete decision chain while reducing the interference of outliers on the learning process.

[0058] When extracting historical state information, historical action information, historical reward information, and historical next state information from preprocessed historical data, each business interaction needs to be broken down into a four-tuple structure of state-action-result-next state in chronological order. Historical state information can be composed of environmental feature vectors at the current decision-making moment, such as a combination of fields including customer profile, historical behavior summary, current request context, and risk assessment results. Historical action information corresponds to the decision action selected by the system at that time; in discrete scenarios, this can be an action number, and in continuous scenarios, it can be a vector form of multiple control variables. Historical reward information comes from key indicators in the business execution results, such as the benefits brought by this decision, risk losses, default occurrence markers, degree of improvement in medical health, and user feedback ratings. These indicators can be synthesized into a single scalar reward through rule mapping or numerical functions. Historical next state information is constructed based on the environmental records of the next moment or stage. It can be retrieved from subsequent logs for the next record of the same object, and its environmental features can be re-encoded to form a vector with the same structure as the historical state information. Through the above extraction process, multiple consecutive logs can be chained together into multiple adjacent state transition segments in the time dimension, with each segment using a set of four-tuples to completely represent a decision and its consequences.

[0059] When constructing an offline dataset based on historical state information, historical action information, historical reward information, and historical next state information, it is necessary to organize a large number of four-tuple samples into an efficient data structure that can be read for training. The offline dataset can be stored using key-value pairs, sequential files, columnar storage, or a dedicated experience replay buffer. Each sample unit contains a state vector, action representation, reward value, and next state vector. To improve training efficiency, samples can be shuffled and packaged in batches, mixing samples from multiple time periods, multiple users, and multiple business scenarios to form a representative experience set. Alternatively, sampling weights can be set for different types of samples according to business strategies, such as increasing the proportion of samples from high-risk scenarios, abnormal events, and key indicator fluctuation ranges, so that the policy network can obtain more learning signals in these sensitive areas. After the offline dataset is constructed, metadata such as the total number of samples, feature dimensions, action dimensions, and reward value range needs to be recorded for direct use during the training environment configuration phase.

[0060] The initialization process of an offline reinforcement learning training environment includes allocating computing resources, selecting devices, setting random number seeds, and establishing log recording channels. The training environment needs to be able to efficiently load offline datasets and support batch iterative reading. Specifically, model instances can be created on GPUs or dedicated accelerated hardware, offline datasets can be mapped to high-throughput storage media, and a stable data flow can be maintained through a queuing mechanism between the data loading process and the training process. Environment initialization also includes defining the loss function expression, optimizer type, gradient pruning rules, and learning rate scheduling strategy to ensure that the calling relationships and parameter update order between modules remain consistent during training.

[0061] The network structure and training hyperparameters of the offline reinforcement learning module determine the expressive power of the policy network and the convergence characteristics of the learning process. The network structure can employ multilayer perceptrons, convolutional structures, attention structures, or combinations thereof, to transform features from state inputs and generate value estimates or policy distributions for different actions. When configuring the network structure, parameters such as the number of hidden layers, the number of neurons per layer, the type of activation function, and the regularization method need to be determined based on the dimensionality of historical state information, the type of action space, and the business's tolerance for complexity. Training hyperparameters include the learning rate, batch size, number of training epochs, target network update frequency, discount factor, and weight decay coefficient. These parameters can be selected through pre-experiments or empirical rules, or tuned before actual deployment using grid search or Bayesian optimization.

[0062] When training a policy network using conservative Q-learning on an offline dataset via an offline reinforcement learning module, constraints on the distribution of action data need to be introduced into the loss function. Conservative Q-learning addresses the inconsistency between policy and action distributions in offline scenarios by adding a penalty term to the objective function. This causes the policy network to produce lower value estimates in action regions not fully covered by data, thus reducing the risk of overestimation. In implementation, a two-part loss can be constructed: a Q-value regression loss based on Bellman residuals and a conservative regularization term addressing the difference between the current policy output and the action data. The weights of these two terms are controlled by adjusting a balancing coefficient. A behavior cloning regularization term further strengthens this constraint, using actions in the action data as a soft reference to guide the policy network to approach historical behaviors in the early stages of training and explore better policies within a safe range in the later stages. The behavior cloning regularization term can take the form of cross-entropy loss, mean squared error loss, or cosine similarity loss, acting between the policy output distribution and the action encoding. By continuously iterating and updating the policy network parameters in multiple training rounds and evaluating the policy performance at fixed intervals, a post-trained decision policy that is stable on offline data distributions and is not prone to over-extrapolation can be obtained.

[0063] For example, to prevent the "discrete data – policy shift" problem, behavioral cloning terms or distribution conservation terms are introduced:

[0064] in, This represents the loss function with parameter θ as the independent variable, used to measure the quality of the current Q-function. During training, it is typically minimized... To update θ; This represents the set of parameters for a Q-function approximator, such as the weights and biases in a neural network. [ ] indicates that the expected value of the quantity in brackets is calculated under the data distribution D. Indicates from empirical dataset Extract state-action pairs (s, a), for example, by sampling from historical decision logs; This represents the set of experience data used for training, corresponding to a batch of historical states, actions, and rewards, constituting an offline data distribution. ( , The expression indicates that the smaller of the two inputs is taken. Here, when comparing two Q values, only the smaller one is kept, making the update more "conservative" and suppressing overestimation. Let θ represent the estimated reward value of the current Q function under state s and action a, and θ be the estimated function to be trained. The reference Q-value can be the target Q-function, the Q-value generated by the baseline strategy, or a target value calculated by the target network, Bellman backpropagation, etc., used for constraint or comparison. .

[0065] This embodiment extracts historical decision log data from the business system and performs systematic preprocessing, transforming the originally scattered and redundant business records into a unified four-tuple of historical state information, historical action information, historical reward information, and historical next state information. This constructs an offline dataset covering real business scenarios, providing sufficient samples for policy learning without introducing additional online exploration risks. By initializing a dedicated offline reinforcement learning training environment and reasonably configuring the network structure and training hyperparameters, the policy network can efficiently fit the actual complexity of the state space and action space. By adopting a conservative Q-learning approach and introducing a behavior cloning regularization term during training, the value extrapolation bias and the tendency to amplify unsafe actions in offline scenarios are effectively suppressed, making the trained decision policy closer to the historical safe behavior trajectory while achieving performance improvement within the range supported by the data.

[0066] In one embodiment, step S30 above includes: S301, loads the encapsulated agent framework containing the basic model module and policy interface module; S302, Deploy the trained decision-making strategy to the policy interface module of the intelligent agent framework; S303, Establish a communication connection between the strategy interface module and the basic model module; S304, Configure the input and output interfaces of the intelligent agent framework to form an integrated intelligent agent system; S305, Perform functional verification on the integrated intelligent agent system; S306, When the functional verification is successful, the integrated agent system is marked as a deployed agent.

[0067] In this embodiment, when loading the encapsulated agent framework containing the basic model module and the policy interface module, it is necessary to retrieve the encapsulated agent structure from the storage area of ​​the computing environment. This structure contains two types of functional units: the basic model module and the policy interface module. The former undertakes the inference tasks of feature understanding and decision interpretation generation, while the latter is responsible for receiving external input, invoking decision policies, and outputting action sets. The loading process is typically completed through the framework initialization function, which internally constructs memory mapping, loads weight files, registers internal routing tables, and allocates a separate parameter area in memory for the policy interface module to ensure that the default policy parameters can be directly written to and overwritten when deploying the trained decision policies.

[0068] When deploying the trained decision policy to the policy interface module of the agent framework, the parameter set output during the training phase needs to be written in a format consistent with that of the policy interface module. The parameter set may include network layer weight matrices, bias terms, attention structure parameters, and hidden state initialization parameters. The deployment process can use either model serialization or structured tensor format, written into the policy interface module through a model loading function or weight replacement function. This allows the policy interface module to generate a probability distribution of decision actions or a value assessment result based on the trained decision policy when receiving input state information. During this process, it is necessary to verify whether the trained decision policy is consistent with the computational operator version, dimension configuration, and feature encoding method used by the current agent framework. If they are inconsistent, format conversion and dimension adaptation are required before deployment to avoid situations where inference is impossible or biased after deployment.

[0069] When establishing a communication connection between the policy interface module and the base model module, a data path needs to be set up so that the policy interface module can send the action encoding to the base model module after generating the decision action, and simultaneously pass the current state vector to the base model module so that the base model module can generate semantic information for interpretation. The communication connection generally includes data pointer mapping between structures, function call binding across modules, shared memory registration, and, when necessary, asynchronous message channel settings. Through this connection, the decision action and the interpretation generation process can be kept synchronized, ensuring that the generated interpretation information remains consistent with the trained decision policy.

[0070] When configuring the input and output interfaces of the intelligent agent framework, it is necessary to set up input and output ports for the intelligent agent system according to the format requirements of the execution environment. The input port is used to receive the environment state vector, and the output port is used to output decision actions and explanation information. The configuration process includes defining the input data type, dimensions, value range, preprocessing method, and the output action structure, explanation text format, and serialization format. In some scenarios, the output interface needs to be further packaged into an API call, enabling external systems to read the actions and explanations in a unified format. After configuration, the basic model module and policy interface module within the intelligent agent framework can form a closed loop, allowing external environment data to trigger policy inference and explanation generation sequentially after input.

[0071] When performing functional verification on the integrated agent system, it is necessary to construct a verification dataset or invoke a simulation environment to check the consistency, stability, and boundary conditions of the input state vector and output results. The verification process not only focuses on whether the trained decision-making strategy can generate actions correctly, but also includes whether the basic model modules can generate matching explanations based on those actions. To ensure the integration effect, the verification phase also checks whether the action format meets business requirements, whether the explanation information covers the key points of the strategy decision, and whether the response time meets the execution environment requirements. If problems such as dimension mismatch, communication connection failure, or inconsistent action explanations occur during functional verification, it is necessary to re-examine the module interoperability or redeploy the trained decision-making strategy.

[0072] Once functional verification is successful, marking the integrated agent system as a deployed agent signifies that the system is ready for use in a real decision-making environment. This marking is typically accomplished by setting internal flags, writing to a configuration file, or registering with the service management layer, enabling the scheduler, external calling system, or business process engine to recognize the deployed agent as an executable instance. After marking, the deployed agent can receive current state information through interfaces in the decision-making environment, invoke trained decision strategies and basic model modules, and output decision actions and corresponding explanations.

[0073] This embodiment loads the encapsulated intelligent agent framework, deploys the trained decision-making strategy, establishes a communication connection between the strategy interface module and the basic model module, configures the input / output interface, and completes functional verification. This enables the trained decision-making strategy to form a collaborative overall system structure with the basic model module, ensuring consistency between the decision-making action and the interpretation and generation process, and providing stability and callability in the actual operating environment.

[0074] In one embodiment, step S40 above includes: S401 obtains real-time environmental data through the decision environment interface; S402, combine the real-time environmental data to form a current state vector; S403, input the current state vector into the policy network of the deployed agent, and generate a decision action probability distribution based on the current state vector through the policy network; S404, Select the optimal decision action based on the probability distribution of the decision action; S405, Input the optimal decision action and the current state vector into the basic model module for deploying the agent; S406, Generate natural language-described decision explanation information through the basic model module; S407, output the optimal decision action and the corresponding decision explanation information.

[0075] In this embodiment, within the decision-making environment, a deployed intelligent agent receives and processes current state information, enabling the system to generate actions and explanatory outputs based on input conditions. The entire process revolves around the acquisition of external data streams, the construction of state vectors, the inference and execution of the policy network, and the generation of explanatory information. When acquiring real-time environmental data through the decision-making environment interface, structured or semi-structured data needs to be read from the input end of the decision-making environment. This data can originate from monitoring modules, business servers, user operation sequences, or sensor acquisition links. The decision-making environment interface can take the form of a function call interface, a data stream port, an API access point, or a message transmission component. Its function is to continuously provide the deployed intelligent agent with the latest environmental features during execution, providing basic input for action inference.

[0076] When combining real-time environmental data to form the current state vector, the original data needs to undergo dimensional unification, anomaly removal, format normalization, and numerical encoding to enable data from different sources to be combined into a vector representation with fixed dimensions. The current state vector typically contains elements such as environmental features, historical feature summaries, business indicator codes, or user behavior signals. Feature preprocessing functions map this information to a continuous vector space, allowing the policy network to directly access it. The construction process usually includes normalization, bucketing encoding, embedding mapping, or temporal compression to ensure that the state representation retains key dynamic information while meeting the input requirements of the policy network.

[0077] When the current state vector is input into the policy network of the deployed agent, it needs to be passed to the computation graph inside the policy network via a tensor format. The policy network may consist of a multi-layered perceptual structure, an attention structure, a value function branch, and a policy branch, etc., to generate a decision action probability distribution based on the current state vector. The action probability distribution reflects the policy network's score or probability value for each executable action, derived from the Softmax mapping of the policy branch or the parameterized distribution calculation of continuous actions. The process of generating the decision action probability distribution based on the current state vector by the policy network relies on computational steps such as weight matrix operations, nonlinear activation, normalization, and probability score generation, while maintaining parameter stability and numerical accuracy internally to ensure that the action distribution does not experience gradient explosion or probability shift.

[0078] When selecting the optimal decision action based on the probability distribution of decision actions, it is necessary to extract the action index or action value from the probability distribution. If the action space is a discrete set, the optimal action is determined by sorting, maximum value selection, or threshold filtering; if the action space is a continuous space, the action vector is obtained through the extreme value solution of the distribution parameters, sampling function, or stability constraint function. The determination of the optimal decision action must simultaneously ensure action executability, action format consistency, and compatibility with the expected output structure of the policy network, so that subsequent modules can directly read it.

[0079] When deploying the agent's base model module by inputting the optimal decision action and the current state vector, the action encoding and state vector need to be merged into a structured input. This allows the base model module to generate explanatory content based on behavioral and state information. The base model module typically consists of a sequence reasoning structure or a multimodal embedding structure, internally analyzing the reasons for action decisions through attention computation, semantic matching, and context decoding. The input action information and input state information together constitute the context for interpretive generation, ensuring that the generated text corresponds to the current action and environmental conditions.

[0080] When generating natural language-based decision explanations using the basic model module, the internal semantic structure needs to be decoded to ensure a clear logical structure. The generation process typically involves steps such as embedded queries, semantic alignment, language modeling, and decoding control. This ensures that the explanations not only explain the reasons for action choices but also reflect the environmental factors that the policy network focuses on, guaranteeing readability and business consistency. The decoded explanations are output as text sequences, and the integrity of the output can be ensured through punctuation control, sequence truncation, and semantic rearrangement.

[0081] When outputting the optimal decision action and its corresponding explanation, both the action and explanation must be sent simultaneously to the output port of the decision environment. The output interface can use structured objects, text messages, API response bodies, or business instruction formats, enabling external systems to synchronously receive the action and explanation. The output behavior is formatted to complete the entire process, creating a unified output structure for the action reasoning and explanation generation stages, facilitating direct invocation in various execution systems.

[0082] This embodiment receives and processes current state information within the decision-making environment, enabling the deployed agent to generate actions and explanations based on the latest environmental conditions, thus achieving integrated output of automatic decision-making and automatic explanation. The acquisition of real-time environmental data, the construction of state vectors, the generation of action probability distributions, and the generation of explanation text form a tight data flow, improving decision consistency, explanation transparency, and execution controllability.

[0083] In one embodiment, step S50 above includes: S501, perform policy matching verification with the preset compliance policy library to generate policy matching verification results; S502, perform content analysis on the decision interpretation information and generate content analysis results; S503, the compliance analysis module analyzes the overall compliance based on the policy matching verification results and the content analysis results, and generates a comprehensive compliance analysis result; S504, Generate a compliance review report based on the comprehensive compliance analysis results; S505, Determine the compliance review result based on the aforementioned compliance review report; S506, The compliance review results are sent to the decision execution system.

[0084] In this embodiment, when conducting compliance reviews of decision-making actions and explanations, a systematic verification of the behavioral output and explanation content is required to ensure that the decision results meet regulatory requirements, business rules, and ethical constraints before actual execution. During execution, the decision-making actions are matched against a pre-defined compliance strategy library. This requires first reading structured data from the strategy library, including rule expressions, behavioral constraints, threshold limits, and a list of prohibited actions, and then comparing the action fields using a matching engine. Strategy matching verification is typically implemented based on rule matching functions, condition tree structures, Boolean expression evaluation functions, or lightweight inference operators, enabling each action to be mapped to an entry in the strategy library, thereby generating strategy matching verification results. The strategy matching verification results include information such as successful matching, failed matching, rule hit level, or violation type label, which are used in subsequent analysis stages.

[0085] Content analysis of decision explanation information requires parsing the explanatory text sequence. Content analysis typically includes text segmentation, syntactic structure extraction, keyword extraction, sensitive word detection, and pronoun reference identification to identify potential inaccuracies, ambiguities, biased word choices, or non-compliant statements within the explanatory information. The results of content analysis can be generated using classification models, logical rule sequences, or semantic mapping structures, enabling the explanatory text to be converted into compliance feature vectors, thus providing computational input for compliance analysis.

[0086] When analyzing overall compliance based on strategy matching verification results and content analysis results using the compliance analysis module, the two input results need to be combined within a unified analysis space. Internally, the compliance analysis module uses feature merging functions, weighted fusion structures, or decision tree structures to jointly complete a comprehensive judgment from both the rule compliance dimension and the explanatory content dimension. This module typically includes scoring functions, violation level calculation rules, comprehensive judgment logic, or attribution generation structures, enabling results from different sources to form a comprehensive compliance analysis result within a unified framework. The comprehensive compliance analysis result is represented in a structured form, including risk level, violation type, and audit confidence level, serving as the basis for subsequent report generation.

[0087] When generating a compliance review report based on the results of a comprehensive compliance analysis, the judgment content needs to be constructed into readable text or structured data objects according to a predetermined format. A compliance review report typically includes audit conclusions, the basis for the judgments, reference rule numbers, a summary of the explanatory analysis, and optional improvement suggestions. A report generator transforms internal analysis results into an externally understandable expression, enabling the decision-making and execution system to read and display the audit conclusions.

[0088] When determining the compliance review result based on the compliance review report, it is necessary to extract the review conclusion field from the report. This process can be achieved by converting the report's structured fields into executable tags, such as "Approved," "Rejected," and "Further Review Required," using mapping rules. This mapping process is accomplished using logical judgment functions, with the final tags serving as the trigger conditions for invoking execution actions.

[0089] When sending compliance review results to the decision-making execution system, the review conclusions need to be pushed to the execution system via a communication interface or callback function. The sending process can be based on message queues, service call interfaces, or data pipelines, enabling the execution system to select subsequent execution paths based on the review conclusions, such as performing actions, pausing execution, or triggering supplementary verification processes. The entire transmission process must maintain data format consistency so that the review results can be directly read by the execution system without additional conversion.

[0090] This embodiment enables the system to perform a two-layer verification of both the decision-making actions and their explanations before execution, forming a complete chain from rule verification to explanation content inspection and comprehensive judgment. This process, through policy matching verification, text analysis, comprehensive evaluation, and report generation, ensures that the pre-execution behavioral output is auditable and traceable, reducing the occurrence of violations, improving decision transparency and explanation credibility, and thus enhancing the security and consistency of decision output in highly sensitive business scenarios.

[0091] In one embodiment, after step S60 above, the method further includes: S701, Generate current reward information based on the decision result; S702, the current state information, the decision action, the current reward information, and the next state information generated after executing the decision action are stored as new samples in the cache dataset; S703, update the offline dataset using the cached dataset, and retrain the decision policy based on the updated offline dataset.

[0092] In this embodiment, when the compliance review is passed, the system needs to trigger the execution chain without additional manual intervention. Therefore, after the compliance review module outputs the compliance review result, the decision execution unit judges the status field in the compliance review result. When the status field meets the pass condition, the decision action issuance process is triggered. If the status field is fail or requires manual confirmation, the subsequent execution process is blocked, no reward is generated, and no data is written to the cached dataset. This ensures that only decision actions that meet the compliance requirements enter the reinforcement learning feedback loop.

[0093] When executing decision actions and obtaining decision results, the decision execution unit converts the action parameters into a call format recognizable by the business system, based on the action type, target object identifier, and parameter set encapsulated in the decision action. The execution process can send instructions to the business system via remote API calls, message queues, or workflow engines. The business system then performs specific business operations for scenarios such as policy adjustments, claims processing, changes to customer benefits, and health management plan push notifications. After execution, the business system returns business execution output, which includes success / failure markers, key numerical indicators, timestamps, exception information, and updated business status. The decision execution unit extracts fields relevant to reinforcement learning from the business execution output as decision results, such as changes in premium income, payout amounts, customer response behavior, and the completion status of health intervention plans.

[0094] When generating current reward information based on decision results, the decision results need to be mapped into numerical reward amounts. The reward calculation process relies on a pre-configured reward definition structure, which records the correspondence between business-side goals and numerical indicators. For example, in a financial scenario, increased premium income, reduced default probability, and decreased complaint rate can be mapped as positive rewards; in a healthcare scenario, improved patient compliance, reduced risk events, and improved health assessment scores can be mapped as positive rewards. The reward calculation process is completed through a set of functions, combining multiple indicators through weighted summation, piecewise functions, or nonlinear functions to obtain the current reward information. The current reward information not only includes a single scalar value but can also include multi-dimensional components after decomposition, such as revenue-dimensional rewards, risk-dimensional rewards, and experience-dimensional rewards, providing expansion space for subsequent algorithm selection.

[0095] After generating the current reward information, the current state information, decision action, current reward information, and the next state information generated after executing the decision action need to be combined into a new sample. The current state information comes from the state vector obtained and encoded by the agent through the environment interface before the current round of decision-making. The decision action is the action identifier and parameters output by the policy network and confirmed by the execution module. The next state information can be encoded through the latest business state field returned by the business system and data re-collected through the environment interface, such as feature vectors after policy status updates, feature vectors after user behavior events are triggered, and time series embeddings after health indicator monitoring. The new sample is organized into a four-tuple or extended structure according to a unified format, and the current state information, decision action, current reward information, and next state information are packaged into a single record and written to the cache dataset. The cache dataset is generally built based on a high-performance storage structure, which can use a memory queue, circular buffer, or lightweight key-value storage to store recently generated new samples, and includes metadata such as timestamps, decision batch numbers, and policy version numbers for easy subsequent filtering and tracking.

[0096] When updating the offline dataset using the cached dataset, the training management module will extract a batch of new samples from the cached dataset according to a preset update strategy when trigger conditions are met in the time dimension, sample quantity dimension, or quality dimension. The update strategy can be based on a simple append method, directly merging the new samples to the end of the offline dataset; it can also resample old and new samples based on sampling weights to control the stability of the data distribution; or it can combine eviction rules when space resources are limited, replacing samples that are too old or have low value contribution. The offline dataset is usually stored in data tables, sequence files, or compressed binary formats. During updates, it is necessary to maintain the consistency of the state information encoding method, action encoding method, and reward scale to avoid shifts in the feature space between different batches of samples.

[0097] After updating the offline dataset, the training management module retrains the decision policy based on the updated offline dataset. The retraining process can be either a complete retraining approach, training the policy network on an expanded dataset from the initial parameters, or an incremental training approach, iteratively refining the policy based on the current network parameters and adjusting the policy with new samples. The training process follows the offline reinforcement learning paradigm, combining historical state information, historical action information, historical reward information, and historical next-state information, utilizing a conservative Q-learning structure, behavioral cloning regularization, or other stability constraints to optimize the objective and iteratively update the policy network. Upon completion of training, the system generates a new version of the decision policy and registers information such as the version number, training time, and dataset range used through the policy version management module.

[0098] When updating the decision-making policy in a deployed agent, the old policy in the deployment environment needs to be replaced with the new policy. This replacement can be done via hot updates, where the parameter weights in the policy interface module are updated to the new version without interrupting online services, and the policy version flag is switched to the latest value after a successful update. To avoid policy jitter, a smooth transition can also be used, where the outputs of the old and new policies are weighted and fused during a transition period, gradually increasing the proportion of the new policy. Throughout the update process, it is necessary to ensure that the basic model modules, state encoding methods, and decision action encoding methods of the deployed agent remain consistent with the training process, enabling the new decision-making policy to operate within the same expression space and achieving a stable transition from offline learning loop to online execution loop.

[0099] This embodiment, by executing decision-making actions after compliance review and introducing a complete closed-loop process including current reward information generation, cached dataset writing, offline dataset updating, and decision strategy retraining and replacement, can transform the business results of each compliant execution into samples that can be used for learning, enabling the agent to continuously absorb the latest business feedback without engaging in high-risk online exploration.

[0100] In one embodiment, a decision-making device based on offline reinforcement learning is provided, which corresponds one-to-one with the decision-making methods based on offline reinforcement learning in the above embodiments. (Refer to...) Figure 3 , Figure 3 This is a schematic diagram of the functional modules of a preferred embodiment of the decision-making device based on offline reinforcement learning of the present invention. The modules include a model building module 10, an offline reinforcement learning module 20, an agent integration module 30, a decision generation module 40, a compliance review module 50, and a decision execution module 60. Detailed descriptions of each functional module are as follows: The model building module 10 is used to obtain a basic model, adjust the basic model based on domain knowledge to obtain an adjusted basic model, encapsulate the adjusted basic model into an agent, and define a state space, action space and reward function for the agent. The offline reinforcement learning module 20 is used to construct an offline dataset containing historical state information, historical action information, historical reward information and historical next state information based on historical decision data, and to train decision strategies based on the offline dataset using the offline reinforcement learning module. The agent integration module 30 is used to combine the trained decision-making strategy with the agent to form a deployment agent; Decision generation module 40 is used to receive and process current state information through the deployed intelligent agent in the decision environment, and output decision actions and corresponding decision explanation information; The compliance review module 50 is used to conduct compliance review on the decision-making action and the decision explanation information. The decision execution module 60 is used to execute the decision action and obtain the decision result when the compliance review is passed.

[0101] In one embodiment, the model building module 10 is specifically used for: Obtain the pre-trained base model; Construct a domain knowledge corpus containing domain-specific knowledge data points; The base model is supervised and adjusted based on the domain knowledge corpus to obtain the adjusted base model. The adjusted basic model is integrated into the agent framework as the core reasoning module of the agent. Define a state space for the agent, which is composed of environmental feature vectors and agent internal state vectors; Define an action space for the agent, the action space comprising a set of discrete or continuous decision actions; Define a reward function for the agent, wherein the reward value is determined by weighting the performance indicators after the decision action is executed.

[0102] In one embodiment, the offline reinforcement learning module 20 is specifically used for: Obtain historical decision log data from the business system; The historical decision log data is preprocessed to obtain preprocessed historical data; Historical state information, historical action information, historical reward information, and historical next state information are extracted from the preprocessed historical data. An offline dataset is constructed based on the historical state information, historical action information, historical reward information, and historical next state information. Initialize the offline reinforcement learning training environment and load the offline dataset; Configure the network structure and training hyperparameters for the offline reinforcement learning module; The offline reinforcement learning module trains the policy network using the conservative Q-learning method based on the offline dataset, and introduces a behavior cloning regularization term during the training process to obtain the trained decision policy.

[0103] In one embodiment, the intelligent agent integration module 30 is specifically used for: Load the encapsulated agent framework that includes the basic model module and the policy interface module; The trained decision-making strategy is deployed to the policy interface module of the intelligent agent framework; Establish a communication connection between the strategy interface module and the basic model module; Configure the input and output interfaces of the intelligent agent framework to form an integrated intelligent agent system; The integrated intelligent agent system is functionally verified. Once the functional verification is successful, the integrated agent system will be marked as a deployed agent.

[0104] In one embodiment, the decision generation module 40 is specifically used for: Obtain real-time environmental data through the decision environment interface; The real-time environmental data is combined to form the current state vector; The current state vector is input into the policy network of the deployed agent, and the policy network generates a probability distribution of decision actions based on the current state vector. Select the optimal decision action based on the probability distribution of the decision actions; The optimal decision action and the current state vector are input into the basic model module for deploying the agent; The basic model module generates natural language-based decision explanation information. Output the optimal decision action and the corresponding decision explanation information.

[0105] In one embodiment, the compliance review module 50 is specifically used for: The decision action is matched and verified against a preset compliance strategy library to generate a strategy matching and verification result. The decision interpretation information is subjected to content analysis to generate content analysis results; The compliance analysis module analyzes the overall compliance based on the policy matching verification results and the content analysis results, and generates a comprehensive compliance analysis result. A compliance review report is generated based on the comprehensive compliance analysis results. The compliance review results are determined based on the aforementioned compliance review report; The compliance review results are sent to the decision execution system.

[0106] In one embodiment, the decision execution module 60 is specifically used for: Current reward information is generated based on the decision results; The current state information, the decision action, the current reward information, and the next state information generated after executing the decision action are stored as new samples in the cache dataset; The offline dataset is updated using the cached dataset, and the decision policy is retrained based on the updated offline dataset.

[0107] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 4 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides determination and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with external clients via a network connection. When the computer program is executed by the processor, it implements the functions or steps of a server-side decision-making method based on offline reinforcement learning.

[0108] In one embodiment, a computer device is provided, which may be a client, and its internal structure diagram may be as follows: Figure 5As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides decision-making and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it implements client-side functions or steps of a decision-making method based on offline reinforcement learning.

[0109] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps: Obtain a base model, adjust the base model based on domain knowledge to obtain an adjusted base model, encapsulate the adjusted base model into an agent, and define a state space, action space and reward function for the agent; An offline dataset containing historical state information, historical action information, historical reward information, and historical next state information is constructed based on historical decision data. Based on the offline dataset, an offline reinforcement learning module is used to train the decision strategy. The trained decision-making strategy is combined with the agent to form a deployment agent; In the decision-making environment, the deployed intelligent agent receives and processes the current state information, and outputs decision actions and corresponding decision explanation information. The decision-making actions and the decision explanation information shall be subject to compliance review; When the compliance review is passed, the decision-making action is executed and the decision result is obtained.

[0110] In one embodiment, a non-volatile computer-readable storage medium is provided, which may be non-volatile or volatile, and stores a computer program thereon. When the computer program is executed by a processor, it performs the following steps: Obtain a base model, adjust the base model based on domain knowledge to obtain an adjusted base model, encapsulate the adjusted base model into an agent, and define a state space, action space and reward function for the agent; An offline dataset containing historical state information, historical action information, historical reward information, and historical next state information is constructed based on historical decision data. Based on the offline dataset, an offline reinforcement learning module is used to train the decision strategy. The trained decision-making strategy is combined with the agent to form a deployment agent; In the decision-making environment, the deployed intelligent agent receives and processes the current state information, and outputs decision actions and corresponding decision explanation information. The decision-making actions and the decision explanation information shall be subject to compliance review; When the compliance review is passed, the decision-making action is executed and the decision result is obtained.

[0111] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and client side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0112] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0113] It should be noted that if any software tools or components not belonging to this company appear in the embodiments of this application, they are merely illustrative examples and do not represent actual use. The embodiments described above are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

[0114] The user personal information involved in this application embodiment is all authorized (knowing and consenting) by the relevant parties or fully authorized by all parties, and the executing entity can obtain it through various open, legal and compliant means. The collection, storage, use, processing, transmission, provision and disclosure of the information, data and signals involved all comply with the relevant laws and regulations of the relevant countries and regions, and do not violate public order and good morals.

Claims

1. A decision-making method based on offline reinforcement learning, characterized in that, Includes the following steps: Obtain a base model, adjust the base model based on domain knowledge to obtain an adjusted base model, encapsulate the adjusted base model into an agent, and define a state space, action space and reward function for the agent; An offline dataset containing historical state information, historical action information, historical reward information, and historical next state information is constructed based on historical decision data. Based on the offline dataset, an offline reinforcement learning module is used to train the decision strategy. The trained decision-making strategy is combined with the agent to form a deployment agent; In the decision-making environment, the deployed intelligent agent receives and processes the current state information, and outputs decision actions and corresponding decision explanation information. The decision-making actions and the decision explanation information shall be subject to compliance review; When the compliance review is passed, the decision-making action is executed and the decision result is obtained.

2. The decision-making method based on offline reinforcement learning as described in claim 1, characterized in that, Obtain a base model, adjust the base model based on domain knowledge to obtain an adjusted base model, encapsulate the adjusted base model into an agent, and define a state space, action space, and reward function for the agent, including: Obtain the pre-trained base model; Construct a domain knowledge corpus containing domain-specific knowledge data points; The base model is supervised and adjusted based on the domain knowledge corpus to obtain the adjusted base model. The adjusted basic model is integrated into the agent framework as the core reasoning module of the agent. Define a state space for the agent, which is composed of environmental feature vectors and agent internal state vectors; Define an action space for the agent, the action space comprising a set of discrete or continuous decision actions; Define a reward function for the agent, wherein the reward value is determined by weighting the performance indicators after the decision action is executed.

3. The decision-making method based on offline reinforcement learning as described in claim 1, characterized in that, An offline dataset containing historical state information, historical action information, historical reward information, and historical next state information is constructed based on historical decision data. Based on this offline dataset, an offline reinforcement learning module is used to train a decision policy, including: Obtain historical decision log data from the business system; The historical decision log data is preprocessed to obtain preprocessed historical data; Historical state information, historical action information, historical reward information, and historical next state information are extracted from the preprocessed historical data. An offline dataset is constructed based on the historical state information, historical action information, historical reward information, and historical next state information. Initialize the offline reinforcement learning training environment and load the offline dataset; Configure the network structure and training hyperparameters for the offline reinforcement learning module; The offline reinforcement learning module trains the policy network using the conservative Q-learning method based on the offline dataset, and introduces a behavior cloning regularization term during the training process to obtain the trained decision policy.

4. The decision-making method based on offline reinforcement learning as described in claim 1, characterized in that, The trained decision-making strategy is combined with the agent to form a deployment agent, including: Load the encapsulated agent framework that includes the basic model module and the policy interface module; The trained decision-making strategy is deployed to the policy interface module of the intelligent agent framework; Establish a communication connection between the strategy interface module and the basic model module; Configure the input and output interfaces of the intelligent agent framework to form an integrated intelligent agent system; The integrated intelligent agent system is functionally verified. Once the functional verification is successful, the integrated agent system will be marked as a deployed agent.

5. The decision-making method based on offline reinforcement learning as described in claim 1, characterized in that, In the decision-making environment, the deployed intelligent agent receives and processes current state information, outputs decision actions and corresponding decision explanations, including: Obtain real-time environmental data through the decision environment interface; The real-time environmental data is combined to form the current state vector; The current state vector is input into the policy network of the deployed agent, and the policy network generates a probability distribution of decision actions based on the current state vector. Select the optimal decision action based on the probability distribution of the decision actions; The optimal decision action and the current state vector are input into the basic model module for deploying the agent; The basic model module generates natural language-based decision explanation information. Output the optimal decision action and the corresponding decision explanation information.

6. The decision-making method based on offline reinforcement learning as described in claim 1, characterized in that, The compliance review of the decision-making actions and the decision interpretation information includes: The decision action is matched and verified against a preset compliance strategy library to generate a strategy matching and verification result. The decision interpretation information is subjected to content analysis to generate content analysis results; The compliance analysis module analyzes the overall compliance based on the policy matching verification results and the content analysis results, and generates a comprehensive compliance analysis result. A compliance review report is generated based on the comprehensive compliance analysis results. The compliance review results are determined based on the aforementioned compliance review report; The compliance review results are sent to the decision execution system.

7. The decision-making method based on offline reinforcement learning as described in claim 1, characterized in that, When the compliance review is passed, after executing the decision-making action and obtaining the decision result, the process further includes: Current reward information is generated based on the decision results; The current state information, the decision action, the current reward information, and the next state information generated after executing the decision action are stored as new samples in the cache dataset; The offline dataset is updated using the cached dataset, and the decision policy is retrained based on the updated offline dataset.

8. A decision-making device based on offline reinforcement learning, characterized in that, The decision-making device based on offline reinforcement learning includes: The model building module is used to obtain a basic model, adjust the basic model based on domain knowledge to obtain an adjusted basic model, encapsulate the adjusted basic model into an agent, and define a state space, action space and reward function for the agent. The offline reinforcement learning module is used to construct an offline dataset containing historical state information, historical action information, historical reward information, and historical next state information based on historical decision data. Based on the offline dataset, the offline reinforcement learning module is used to train the decision strategy. The agent integration module is used to combine the trained decision-making strategy with the agent to form a deployment agent; The decision generation module is used to receive and process current state information through the deployed intelligent agent in the decision-making environment, and output decision actions and corresponding decision explanation information; The compliance review module is used to conduct compliance reviews of the decision-making actions and the decision explanation information. The decision execution module is used to execute the decision action and obtain the decision result when the compliance review is passed.

9. A computer device, characterized in that, The computer device includes a memory, a processor, and an offline reinforcement learning-based decision-making program stored in the memory and executable on the processor, wherein the offline reinforcement learning-based decision-making program, when executed by the processor, implements the steps of the offline reinforcement learning-based decision-making method as described in any one of claims 1-7.

10. A non-volatile computer-readable storage medium, characterized in that, The storage medium stores a decision-making program based on offline reinforcement learning, which, when executed by a processor, implements the steps of the decision-making method based on offline reinforcement learning as described in any one of claims 1-7.

Images