A rule-embedded intelligent policy decision method and system
By embedding policy rules into the model's internal logic and combining supervised learning and reinforcement learning training, the problems of latency, high cost, and insufficient robustness in existing decision-making systems are solved, achieving efficient and reliable intelligent decision support.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- INST OF SOFTWARE - CHINESE ACAD OF SCI
- Filing Date
- 2026-03-30
- Publication Date
- 2026-07-03
AI Technical Summary
Existing intelligent strategy decision-making systems suffer from problems such as lax rule enforcement, poor system latency and real-time performance, high construction and maintenance costs, and limited robustness and adaptability in high-efficiency and high-reliability real-world scenarios, and cannot meet the demands of modern mission-critical tasks for decision-making speed, accuracy, and reliability.
By adopting an embedded rule architecture, the policy rules are deeply internalized into the model's inherent logic. Game adversarial data is obtained through the policy model, and decision schemes are generated using a backbone network, situation encoder, rule constraint layer, and scheme generation layer. The model is trained by combining supervised learning and reinforcement learning to achieve real-time and rigorous decision-making.
It achieves millisecond-level decision generation, significantly improving the standardization and logical rationality of decisions, reducing knowledge maintenance costs, enhancing system robustness and user trust, and meeting the real-time decision-making needs of highly dynamic scenarios.
Smart Images
Figure CN122334474A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of artificial intelligence technology, specifically relating to a rule-embedded intelligent strategy decision-making method and system. Background Technology
[0002] With the deepening application of artificial intelligence technology in complex decision-making scenarios, intelligent strategy decision-making systems have become a key tool for improving operational efficiency and responsiveness. Currently, the core technical approach in this field generally relies on external knowledge enhancement architectures, that is, by integrating knowledge graphs and retrieval-enhanced generation components, domain knowledge and rule constraints are injected into large language models to compensate for their inherent lack of factual basis and logical limitations.
[0003] Typical system designs employ a multi-module integration approach. First, a large structured knowledge base (knowledge graph) needs to be built and maintained, a process involving continuous entity extraction, relationship building, and data fusion. During decision-making, the system retrieves relevant fragments from the knowledge base or rule base using vector retrieval technology. Then, through careful prompting engineering, the retrieved rules and facts are fed as contextual input to the generative model, ultimately producing decision recommendations. Furthermore, existing solutions typically include two main functional components: one is a strategy proposal generation algorithm, which attempts to encode domain norms into the model context through multimodal information input and rule-embedded prompts; the other is auxiliary situational analysis technology, which aims to optimize the decision-maker's analysis process through automatic problem discovery, evidence-driven information summarization, and structured report generation.
[0004] However, such external knowledge augmentation architectures have a series of inherent drawbacks that severely limit their deployment in efficient and highly reliable real-world scenarios: 1. Insufficient strictness in rule enforcement: The system treats decision rules as "knowledge" that can be retrieved from the outside, rather than "rules" that the model must internalize and follow. This separation design leads to the model easily ignoring, distorting, or weakening the retrieved rule constraints under high pressure or complex contexts, failing to fundamentally guarantee that decision-making behavior strictly adheres to established norms.
[0005] 2. System latency and poor real-time performance: The serial processing flow of multiple modules (such as retrieval, suggestion construction, and generation) introduces significant latency, making it difficult to meet the urgent needs of real-time or near-real-time decision-making in highly dynamic environments. The update of knowledge state cannot be synchronized with the rapidly changing scenario, often leading to decisions based on outdated information.
[0006] 3. High construction and maintenance costs: The construction, updating, and expansion of knowledge graphs require continuous investment of professional knowledge and human resources. Any update to rules or knowledge may trigger modifications across the entire chain, from knowledge base reconstruction to prompt template adjustments, resulting in poor agility and high costs.
[0007] 4. Limited Robustness and Adaptability: Under conditions of information noise or adversarial input, the system's decision accuracy deteriorates rapidly. Its performance is highly dependent on the quality and relevance of the retrieved content, essentially shifting the system's reliability risk to the completeness of the external knowledge base and the accuracy of the retrieval.
[0008] In summary, the fundamental problem with existing technological approaches lies in their "internal-external separation" design philosophy—treating policy rules as pluggable external knowledge attachments rather than deeply internalized capabilities within the model's reasoning mechanism. This leads to multiple bottlenecks in the system's rigor, real-time performance, economy, and robustness, failing to meet the triple extreme demands of modern mission-critical tasks for decision-making speed, accuracy, and reliability. Therefore, exploring a new paradigm that enables models to deeply internalize rules and achieve efficient, rigorous, and autonomous decision-making has become a core issue urgently needing breakthroughs in this field. Summary of the Invention
[0009] This invention discloses a rule-embedded intelligent strategy decision-making method and system, which deeply internalizes strategy rules into the model's inherent logic, thereby achieving both real-time and rigorous decision-making while avoiding reliance on external knowledge bases.
[0010] To achieve the above objectives, the technical solution of the present invention includes the following:
[0011] A rule-embedded intelligent strategy decision-making method obtains decision schemes corresponding to game adversarial data based on a strategy model; the strategy model includes: Backbone network, used to generate situation vectors based on game adversarial data; A situation encoder is used to transform situation vectors into a situation representation matrix that reflects spatial relationships and environmental influences. The rule constraint layer is used to generate mathematical constraints based on the situation representation matrix. The scheme generation layer is used to generate decision schemes based on the situation representation matrix and mathematical constraints.
[0012] Furthermore, the training process of the policy model includes: Based on the full text of the basic rules of game adversarial competition, typical scenario descriptions and historical cases, parameter database and environmental effect table, respectively, rule layer sample set, scenario layer sample set and micro layer sample set are constructed; After supervised learning training of the backbone network based on the rule layer sample set, scene layer sample set and micro layer sample set, the situation encoder, rule constraint layer and scheme generation layer are trained by supervised learning based on the situation vector output by the trained backbone network to obtain the initial policy model. The initial strategy model is used as both the master decision model and the adversarial model for reinforcement learning training; wherein the master decision model is used to generate decision schemes, and the adversarial model is used to find vulnerabilities in the master decision model within the scope allowed by the rules. Repeat the supervised learning and reinforcement learning training processes until the total reward of the reinforcement learning training converges, and obtain the trained policy model.
[0013] Furthermore, the backbone network includes a basic Transformer model and an extended Transformer model; The supervised learning training of the backbone network based on rule-layer sample sets, scene-layer sample sets, and micro-layer sample sets includes: The sample from the rule layer is input into the base Transformer model, and the parameters of the base Transformer model are updated by calculating the cross-entropy loss of the rule category and the negative log-likelihood loss of the masked token. The parameters of the base Transformer model are fixed, and the scene layer samples are passed through the base Transformer model and the extended Transformer model in sequence. The parameters of the extended Transformer model are updated by calculating the mean squared error loss of the situation at the time before and after the time and the binary cross-entropy loss of the scheme compliance score. The parameters of the fixed base Transformer model and the front parameters in the extended Transformer model The parameters of each Transformer layer are calculated, and the micro-layer samples are sequentially passed through the basic Transformer model and the extended Transformer model. The mean squared error between the predicted and actual environmental parameters is then used to evaluate the parameters in the extended Transformer model. The parameters of each Transformer layer are updated.
[0014] Furthermore, the situation encoder, rule constraint layer, and scheme generation layer are trained under supervised learning based on the situation vector output by the trained backbone network to obtain an initial policy model, including: Based on the spatial distance between entities and the environmental attenuation coefficient, a position code is generated, and based on the situation vector and the position code, a situation characterization matrix is obtained. Extract the threat and resource features involved in each rule from the situational characterization matrix, and calculate the rule constraint loss of the rule based on the threat and resource features. The rule constraint loss of all rules is aggregated, and the parameters of the situation encoder, rule constraint layer and scheme generation layer are updated based on the aggregation result and task failure loss.
[0015] Furthermore, in the case where the rule is a resource redundancy rule, the step of calculating the rule constraint loss based on the threat characteristic and the resource characteristic includes: Based on the threat characteristics and resource characteristics, calculate the threat intensity, the area to be covered, and the amount of available resources. By using independent multilayer perceptrons, the threat intensity and the area to be covered are mapped to resource requirements, and the constraint violation degree is obtained by weighted summation with the amount of available resources. The constraint violation degree is compared with a set threshold, and the rule constraint loss of the rule is obtained based on the larger one.
[0016] Furthermore, the calculation process for the total reward includes: Obtain the goal completion rate, resource utilization rate, time constraint satisfaction rate, severity of rule violation, and importance weight of each rule during the reinforcement learning process; Calculate task channel rewards based on the target completion rate, resource utilization rate, and time constraint satisfaction. Calculate the penalty for the constraint channel based on the severity of each rule violation and the weight of each rule's importance; Subtract the penalty for the restricted channel from the reward for the task channel to get the total reward.
[0017] A rule-embedded intelligent strategy decision-making system, the system comprising: Backbone network, used to generate situation vectors based on game adversarial data; A situation encoder is used to transform situation vectors into a situation representation matrix that reflects spatial relationships and environmental influences. The rule constraint layer is used to generate mathematical constraints based on the situation representation matrix. The scheme generation layer is used to generate decision schemes based on the situation representation matrix and mathematical constraints.
[0018] A computer device includes: a processor and a memory storing computer program instructions; the processor, when executing the computer program instructions, implements the rule-embedded intelligent policy decision-making method described in any of the preceding claims.
[0019] A computer-readable storage medium storing computer program instructions, which, when executed by a processor, implement the rule-embedded intelligent policy decision-making method described in any of the preceding claims.
[0020] A computer program product, when run on a computer device, causes the computer device to execute the intelligent policy decision-making method embedded in the rules described in any of the preceding claims.
[0021] Compared with the prior art, the present invention has at least the following beneficial effects.
[0022] This invention reduces decision generation time to milliseconds through an embedded rule architecture, speeding up the process by three orders of magnitude compared to traditional knowledge graph and retrieval-enhanced generation schemes, fully meeting the stringent real-time decision-making requirements of highly dynamic scenarios. Regarding rule compliance, the violation rate is effectively suppressed, fundamentally solving the problem of lax rule enforcement, especially significantly improving decision stability under high-pressure environments. Knowledge maintenance costs are significantly reduced; rule updates require only hourly model fine-tuning to adapt, avoiding the high cost of rebuilding the entire knowledge base. The system maintains high consistency during continuous decision-making, effectively avoiding decision failures caused by asynchronous knowledge states through a dynamic scene perception mechanism. Thanks to the adversarial reinforcement learning verification framework, this invention maintains stable decision accuracy even in noisy environments, demonstrating outstanding robustness. Simultaneously, the enhanced interpretability of the decision-making process significantly increases user trust in the system, providing a solid foundation for human-machine collaborative decision-making. Compared to existing technologies, this invention significantly reduces decision latency by eliminating external retrieval steps, enhances adaptability to complex environments through adversarial training mechanisms, and enables edge device deployment using knowledge distillation technology, providing a more direct and efficient solution for the field of intelligent decision-making. Attached Figure Description
[0023] Figure 1 A schematic diagram of policy model training. Detailed Implementation
[0024] The present invention will be further described below with reference to possible accompanying drawings and specific embodiments, but this does not constitute any limitation on the present invention.
[0025] This invention presents an intelligent policy decision-making method with embedded rules. By directly encoding policy rules into the structural constraints and training objectives of a neural network, it achieves deep integration of rules and deep learning models, generating policy solutions that meet regulatory requirements without the need for external knowledge retrieval. The core technologies include a formal representation method for policy rules, a neural rule fusion training mechanism, a dynamic scene-adaptive optimization algorithm, and a reliable policy decision verification framework. This invention is applicable to intelligent decision-making systems, simulation training platforms, and auxiliary decision-making systems, and is particularly suitable for collaborative policy scenarios with high timeliness and compliance requirements. It can significantly improve the regulatory compliance, logical rationality, and execution reliability of decisions, providing efficient and reliable support for modern intelligent decision-making.
[0026] This invention is based on a strategy model to obtain decision schemes corresponding to game adversarial data; the strategy model includes: a backbone network for generating situation vectors based on game adversarial data; a situation encoder for converting the situation vectors into a situation representation matrix reflecting spatial relationships and environmental influences; a rule constraint layer for generating mathematical constraints based on the situation representation matrix; and a scheme generation layer for generating decision schemes based on the situation representation matrix and mathematical constraints.
[0027] Figure 1 This diagram illustrates the training of the policy model, which is implemented through two training phases: supervised learning and reinforcement learning. The supervised learning phase is used to internalize rule knowledge, while the reinforcement learning phase optimizes the game-playing scenario to improve the model's robustness.
[0028] (a) Supervision and learning phase.
[0029] The supervised learning phase involves constructing training data to enable the model to master general rule knowledge and acquire basic decision-making capabilities. This phase includes six steps: data preprocessing, rule-level training, scenario-level training, micro-level training, loss aggregation and monitoring, and rule internalization training.
[0030] ① Data preprocessing.
[0031] The data preprocessing step receives three types of raw data as input: basic data, which is the full text of the basic rules of the game; scenario data, which is the description of typical scenarios and historical cases; and micro data, which is the parameter library and environmental effect table.
[0032] First, the basic data undergoes text denoising, removing formatting marks and redundant spaces, and a sequence completion sample set is constructed, where each sample contains a pair of context text and target text. Next, the scene data undergoes consistency verification, checking the logical consistency between scene descriptions and solution annotations, and a sample set of scene-solution pairings is constructed. Finally, the micro-data undergoes numerical normalization, mapping weapon performance parameters, resource consumption rates, terrain influence coefficients, and other values to a unified range, and a sample set of environmental conditions paired with optimal parameters is constructed. Ultimately, a three-tiered sample set is formed.
[0033] ② Rule-based training.
[0034] The rule layer training step receives a first-level sample set and a base Transformer model as input. The model employs a 12-layer structure with randomly initialized parameters. This step inputs the contextual text from the first-level sample set into the base model, predicts the rule category using a classification head and calculates the cross-entropy loss, and simultaneously predicts the masked token using a masking language model head and calculates the negative log-likelihood loss. Then, the two losses are weighted and summed with weights of 0.4 and 0.6 respectively to obtain the rule layer loss. The AdamW optimizer is used to update the model parameters using gradient descent until the loss converges. The rule layer training outputs the rule layer model and its parameters, as well as the rule layer loss value. .
[0035] ③ Scene-level training.
[0036] The scene layer training receives the rule layer model and the secondary sample set as input. First, the rule layer Transformer model is expanded to a 24-layer structure, with 12 new layers having their parameters randomly initialized while the original 12 layers remain fixed. Then, scene data from the secondary sample set is input into the expanded model. The prediction head predicts the situation at the next time step and calculates the mean squared error loss. Simultaneously, the scene and solution pairing is input into the evaluation head to predict compliance scores and calculate the binary cross-entropy loss. The two types of losses are weighted and summed with weights of 0.6 and 0.4 respectively to obtain the scene layer loss. A small learning rate is used to fine-tune the parameters of the new layers, while the original layer parameters remain unchanged, until the loss converges. During this process, the model understands the application logic of the rules in the specific scene. The scene layer training step outputs the scene layer model and its parameters, as well as the scene layer loss value. .
[0037] ④ Micro-level training.
[0038] The micro-layer training step receives the scene layer model and a three-level sample set as input. First, the parameters of the first 18 layers of the scene layer model are fixed, retaining only the top 6 layers for training. The environmental conditions from the three-level sample set are input into the model, and the optimal parameter prediction values are output through the parameter prediction head. Then, the mean squared error between the prediction and the true values is calculated as the micro-layer loss. The top-level parameters are fine-tuned using a minimal learning rate until the loss converges. The micro-layer training step outputs the micro-layer model and its parameters, i.e., the final pre-trained model parameters, and the micro-layer loss value. .
[0039] ⑤ Loss aggregation monitoring.
[0040] The system receives a weighted sum of the rule layer loss, scene layer loss, and micro-level loss, calculates the overall pre-training monitoring metrics, and uses these metrics to determine the model's training status, convergence trend, and stability, thereby improving pre-training efficiency and final model performance. The objective function is designed as follows: in α =0.5、 β =0.3、 γ =0.2 represents the weight coefficients, corresponding to the three dimensions of rule compliance, scenario adaptability, and detail rationality. The overall pre-training monitoring metrics do not participate in gradient backpropagation. After the pre-training phase is completed, the final pre-trained model parameters will be used for initialization in the rule internalization training phase.
[0041] In another embodiment, the present invention can also adjust the three-level training order, for example, prioritizing micro-parameter training before extending to macro-principle learning, or dynamically adjusting the weight coefficients in the pre-trained objective function to adapt to different scenario requirements.
[0042] ⑥ Internalization training of rules.
[0043] During rule internalization training, this invention first constructs a situation encoder and designs an improved Transformer neural network architecture. A dedicated position encoding method is added to the original Transformer model to fuse spatial distance and environmental features. This encoding function is defined as follows: in, For entities and entity Spatial distance between them The environmental attenuation coefficient is set to 1 for open areas and 0.6 for complex areas. The improved network architecture receives standardized situation vectors and location codes as inputs and outputs a situation characterization matrix. This design enables the model to understand the impact of the environment on the behavior of entities.
[0044] Then, construct a rule constraint layer, from Threat and resource features are extracted, and constraint violation degree is calculated for each rule. Its design aims to transform unstructured rules into mathematical constraints that the model can compute and optimize, ensuring that the generated results conform to specifications. Taking resource redundancy rules as an example, the following calculation is performed: in, These are the weighting coefficients. Threat intensity, abbreviated as . The area to be covered, abbreviated as . The amount of resources available to us, abbreviated as: Resources can be categorized into several types. Indicates the number of resource types. Through an independent multilayer perceptron Mapping threat intensity and area of effect to resource requirements describes whether current resources can meet the needs of the threat intensity and mission area of effect, and uses three... Coefficients adjust the importance of threats, regions, and resources: like A value greater than 0 indicates that available resources are insufficient to meet the demands corresponding to the current threat intensity and geographical scope, violating the "resource redundancy" rule and triggering constraint losses. This will be suppressed during model training.
[0045] Construct in this way Each rule constrains the loss and aggregates them to obtain... Specifically, a correlation weight coefficient is designed for the loss value of each rule. This is to demonstrate the importance of the rules.
[0046] Finally, a solution generation layer is constructed. This layer is an end-to-end decision generation module built upon an improved Transformer architecture. Located at the decision output end of the model architecture, it employs a layered decoding strategy. First, it generates a solution outline; then, it refines specific actions; and finally, under rule constraints and situational information limitations, it automatically generates a decision solution. A training loss function is designed. in, It is a loss constrained by rules. It represents the loss from task failure, used to quantify the extent to which a solution cannot achieve its objectives. The model will be guided to generate decision-making solutions that meet the rule constraints and effectively achieve the objectives.
[0047] In another embodiment, the present invention may also use a graph neural network instead of a Transformer encoder in terms of rule embedding architecture, so as to better preserve the logical relationship between rules through a node embedding mechanism.
[0048] (ii) The reinforcement learning stage.
[0049] In the reinforcement learning phase, this invention constructs a game-theoretic adversarial environment to enhance the model's decision robustness and rule compliance through dynamic interaction. This phase employs a dual-model adversarial mechanism: the primary decision model is the initial decision model obtained after rule internalization training, while the adversarial model is initialized from the primary decision model; both models have identical structure and parameters. The primary decision model generates compliant solutions, while the adversarial model assists in improving the robustness of the primary model by identifying vulnerabilities within the permitted rule range. The primary decision model, as the system's output entity, further enhances rule internalization and robustness through self-play.
[0050] The core of the reinforcement learning phase is the design of a dual-channel reward mechanism: one is the task-based reward. This is used to quantify the degree to which the solution achieves its core objectives; the higher the task completion rate, the larger this value. Secondly, it constrains channel rewards. This value quantifies the severity of a rule violation; the more severe the violation, the higher the value. (Task channel reward) It is composed of a weighted average of goal completion rate, resource efficiency, and time efficiency: in, The percentage of the target achieved. For resource utilization, This represents the time constraint satisfaction level.
[0051] Constraint channel penalty Sum the products of the severity of each rule violation and its importance weight: in To determine the severity of the violation of rule i, This represents the importance weight of the rule.
[0052] The total reward is obtained by subtracting the constraint channel penalty from the mission channel reward: in The penalty coefficient is used to impose an exponential penalty on serious rule violations, ensuring the model deeply understands the importance of the rules. Training employs the PPO algorithm for policy updates, achieving stable optimization through advantage function estimation and policy ratio pruning. A progressive learning strategy is also used, gradually increasing scenario complexity, enabling the master decision model to learn to optimize solution generation without violating the rules.
[0053] The reinforcement learning phase and the supervised learning phase form a closed loop. By updating the policy, the optimized parameters are returned to the supervised learning phase, and the learning process can be executed cyclically to bring the total reward to converge.
[0054] In another embodiment, the present invention may also use the Monte Carlo tree search algorithm to replace the PPO reinforcement learning strategy, or introduce a rule engine to replace the adversarial model for vulnerability detection.
[0055] It should be noted that this invention can be applied to multiple scenarios, including drug development, traffic management, and autonomous driving. For example, in an autonomous driving scenario, the adversarial data includes the vehicle's own state data (position, speed, acceleration, heading angle, steering angle, throttle / brake opening, current planned trajectory, target lane, target speed, traffic light status, etc.), other party behavior and intention data (multiple possible trajectories of other vehicles / pedestrians in future time periods and the probability of each trajectory, etc.), environmental and scenario context data (lane lines, curbs, traffic signs, traffic light locations, drivable areas, speed limits of the current road segment, whether lane changing is allowed, stop lines, yield rules, etc.), and interaction and game process data (spatial relationships and interaction dependencies between participants, timing of strategic interactions or conflicts, etc.). Finally, a driving plan for the vehicle is generated based on the decision-making scheme.
[0056] Specifically, this invention addresses the complex game-theoretic problem faced by autonomous vehicles in real-world road environments by constructing a two-stage Transformer backbone network. The base model receives full observation data from sensor fusion, including the vehicle's state, the position, speed, and heading of all traffic participants (vehicles, pedestrians, bicycles, etc.), as well as structured information such as lane lines and traffic signals provided by high-precision maps. The extended model specifically receives refined control command sequences. The situation encoder combines high-precision map coordinates, traffic rules (such as right-of-way), and the kinematic characteristics of participants (such as braking distance and safety radius) to construct a spatiotemporal interaction topology matrix, quantifying the mutual influence relationships between different entities.
[0057] The training phase consists of three steps: Supervised Pre-training: First, supervised learning is performed using a large amount of high-quality human driving data (driving record videos and corresponding operations) to enable the model to initially grasp macro-level driving strategies (such as following other vehicles, lane changing, and intersection handling) and micro-level control sequences (such as smooth transitions between steering and acceleration / deceleration). Rule and Safety Constraint Injection: For hard constraints such as "safe distance," "traffic rule compliance" (such as stopping at red lights and yielding), and "ride comfort," calculable constraint violation rates are designed. For example, when the system generates a strategy that gets too close to a pedestrian, a "safety margin loss" is calculated based on relative speed and distance and backpropagated, forcing the model parameters to be updated to avoid such behavior. Adversarial Reinforcement Learning: An adversarial model is introduced to simulate high-risk traffic participants (such as aggressively cutting in or pedestrians suddenly crossing the road). The main model generates driving strategies within the limits allowed by physical rules and traffic regulations, and its total reward is calculated by a composite function: Total Reward = Task Path Reward - Constraint Path Penalty. The task lane rewards come from efficiently completing driving objectives, such as: smoothly passing through intersections, safely arriving at the route point within the specified time, and efficiently passing through congested areas; the constraint lane penalties are the weighted sum of violations of various safety and rule constraints, such as: running red lights, discomfort caused by sudden braking, and keeping too close to other vehicles.
[0058] During the inference phase, the model receives real-time sensor data streams and generates a current traffic situation vector. The rule constraint layer verifies the legality of decisions in real time, including traffic regulation compliance (whether solid lines are crossed), vehicle dynamic limits (whether steering is too abrupt), and safety and ethical principles in the current scenario. Finally, the solution generation layer outputs a set of trajectory planning and control instructions that comply with all safety norms and traffic regulations within milliseconds, achieving low-latency, high-reliability, fully autonomous decision-making from environmental perception to vehicle execution.
[0059] In summary, against the backdrop of rapid development in artificial intelligence, this invention breaks through the limitations of the traditional "knowledge retrieval-generation" model and proposes an innovative "rule embedding" architecture, directly encoding the basic rules of the strategy into the model's intrinsic capabilities. Technically, this invention integrates deep learning, reinforcement learning, and domain expertise to construct an end-to-end decision support framework, solving key problems such as insufficient robustness and high decision latency in existing systems under dynamic environments, and providing a new technical path for future intelligent decision-making systems.
[0060] The policy rule embedding architecture of this invention breaks through the traditional external knowledge enhancement model, directly encoding policy rules as the structural constraints and parameter distributions of a neural network. By designing a dedicated neural network layer, the basic rules are formalized into differentiable constraint functions, enabling the model to internalize the rules during the training phase. For example, domain experience is quantified and transformed into hard constraint terms in the loss function, automatically suppressing rule-violating schemes during training. This architecture eliminates the external retrieval stage, significantly reduces decision latency, and significantly improves the strictness of rule enforcement.
[0061] This invention proposes a multi-granularity pre-training mechanism with a three-level pre-training strategy encompassing macro, meso, and micro levels: the rule layer pre-trains on the full text of basic rules to grasp core principles; the scenario layer pre-trains in typical scenarios to understand logical relationships; and the micro layer pre-trains on parameters and environmental data to grasp technical details. The pre-training objective function is designed as a weighted combination to ensure the model fully internalizes rule knowledge. This mechanism enables the model to generate compliant solutions without external retrieval, and knowledge updates only require fine-tuning the model, significantly reducing maintenance costs.
[0062] This invention constructs an adversarial reinforcement learning environment, encoding rules as constraints for the reward function. A dual-channel reward mechanism is designed: the main channel rewards task completion, while the constraint channel penalizes rule violations. Through extensive self-play, the model learns to optimize performance without violating the rules. This framework does not rely on external knowledge verification but internalizes rule awareness through adversarial training, significantly improving the system's robustness in complex environments.
[0063] The above embodiments are merely illustrative of the technical solutions of the present invention and are not intended to limit it. Those skilled in the art can modify or make equivalent substitutions to the above technical solutions based on the concept of the present invention, and such modifications or equivalent substitutions should all be covered within the protection scope of the present invention. The protection scope of the present invention is defined by the claims.
Claims
1. A rule-embedded intelligent strategy decision-making method, characterized in that, Decision-making schemes corresponding to game-playing data are obtained based on a strategy model; the strategy model includes: Backbone network, used to generate situation vectors based on game adversarial data; A situation encoder is used to transform situation vectors into a situation representation matrix that reflects spatial relationships and environmental influences. The rule constraint layer is used to generate mathematical constraints based on the situation representation matrix. The scheme generation layer is used to generate decision schemes based on the situation representation matrix and mathematical constraints.
2. The method according to claim 1, characterized in that, The training process of the policy model includes: Based on the full text of the basic rules of game adversarial competition, typical scenario descriptions and historical cases, parameter database and environmental effect table, respectively, rule layer sample set, scenario layer sample set and micro layer sample set are constructed; After supervised learning training of the backbone network based on the rule layer sample set, scene layer sample set and micro layer sample set, the situation encoder, rule constraint layer and scheme generation layer are trained by supervised learning based on the situation vector output by the trained backbone network to obtain the initial policy model. The initial strategy model is used as both the master decision model and the adversarial model for reinforcement learning training; wherein the master decision model is used to generate decision schemes, and the adversarial model is used to find vulnerabilities in the master decision model within the scope allowed by the rules. Repeat the supervised learning and reinforcement learning training processes until the total reward of the reinforcement learning training converges, and obtain the trained policy model.
3. The method according to claim 2, characterized in that, The backbone network includes a basic Transformer model and an extended Transformer model; The supervised learning training of the backbone network based on rule-layer sample sets, scene-layer sample sets, and micro-layer sample sets includes: The sample from the rule layer is input into the base Transformer model, and the parameters of the base Transformer model are updated by calculating the cross-entropy loss of the rule category and the negative log-likelihood loss of the masked token. The parameters of the base Transformer model are fixed, and the scene layer samples are passed through the base Transformer model and the extended Transformer model in sequence. The parameters of the extended Transformer model are updated by calculating the mean squared error loss of the situation at the time before and after the time and the binary cross-entropy loss of the scheme compliance score. The parameters of the fixed base Transformer model and the front parameters in the extended Transformer model The parameters of each Transformer layer are calculated, and the micro-layer samples are sequentially passed through the basic Transformer model and the extended Transformer model. The mean squared error between the predicted and actual environmental parameters is then used to evaluate the parameters in the extended Transformer model. The parameters of each Transformer layer are updated.
4. The method according to claim 2, characterized in that, Based on the situation vector output by the trained backbone network, supervised learning training is performed on the situation encoder, rule constraint layer, and scheme generation layer to obtain the initial policy model, including: Based on the spatial distance between entities and the environmental attenuation coefficient, a position code is generated, and based on the situation vector and the position code, a situation characterization matrix is obtained. Extract the threat and resource features involved in each rule from the situational characterization matrix, and calculate the rule constraint loss of the rule based on the threat and resource features. The rule constraint loss of all rules is aggregated, and the parameters of the situation encoder, rule constraint layer and scheme generation layer are updated based on the aggregation result and task failure loss.
5. The method according to claim 4, characterized in that, In the case that the rule is a resource redundancy rule, the step of calculating the rule constraint loss based on the threat characteristic and the resource characteristic includes: Based on the threat characteristics and resource characteristics, calculate the threat intensity, the area to be covered, and the amount of available resources. By using independent multilayer perceptrons, the threat intensity and the area to be covered are mapped to resource requirements, and the constraint violation degree is obtained by weighted summation with the amount of available resources. The constraint violation degree is compared with a set threshold, and the rule constraint loss of the rule is obtained based on the larger one.
6. The method according to claim 2, characterized in that, The calculation process for the total reward includes: Obtain the goal completion rate, resource utilization rate, time constraint satisfaction rate, severity of rule violation, and importance weight of each rule during the reinforcement learning process; Calculate task channel rewards based on the target completion rate, resource utilization rate, and time constraint satisfaction. Calculate the penalty for the constraint channel based on the severity of each rule violation and the weight of each rule's importance; Subtract the penalty for the restricted channel from the reward for the task channel to get the total reward.
7. A rule-embedded intelligent strategy decision-making system, characterized in that, The system includes: Backbone network, used to generate situation vectors based on game adversarial data; A situation encoder is used to transform situation vectors into a situation representation matrix that reflects spatial relationships and environmental influences. The rule constraint layer is used to generate mathematical constraints based on the situation representation matrix. The scheme generation layer is used to generate decision schemes based on the situation representation matrix and mathematical constraints.
8. A computer device, characterized in that, The computer device includes: a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, it implements the rule-embedded intelligent policy decision-making method as described in any one of claims 1-7.
9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions, which, when executed by a processor, implement the rule-embedded intelligent policy decision-making method as described in any one of claims 1-7.
10. A computer program product, characterized in that, When the computer program product is run on a computer device, the computer device performs the rule-embedded intelligent policy decision-making method as described in any one of claims 1-7.