Knowledge graph-based confrontation environment reinforcement learning model optimization method and system

Abstract

The invention relates to the technical field of intelligent decision making, in particular to a knowledge graph-based confrontation environment reinforcement learning model optimization method and system. Comprising the following steps: acquiring confrontation environment data, analyzing the confrontation environment data into a state-action feature matrix, and constructing a knowledge graph containing an environment entity and confrontation logic; designing a dynamic optimization strategy of the reinforcement learning model based on the entity relationship of the knowledge graph, and generating constraint conditions of environmental state transition through graph reasoning; encoding constraint conditions into a reward function, embedding the reward function into a reinforcement learning framework, and dynamically adjusting a model topological structure by using an adversarial experience playback mechanism; fusing a logical reasoning path of the knowledge graph and a reinforcement learning strategy network, and constructing a bidirectional information interaction mechanism; the knowledge graph entity relationship is updated according to the real-time adversarial data, a dynamic adversarial strategy library is generated, and the decision-making efficiency and accuracy of the model in a complex adversarial environment are improved. The problems of low reinforcement learning model training efficiency and insufficient strategy dynamic adaptability in a confrontation environment are solved.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent decision-making, and particularly to an optimization method and system for a reinforcement learning model in an adversarial environment based on a knowledge graph. Background Art

[0002] In the prior art, the reinforcement learning model in an adversarial environment mainly relies on a static policy library and environment preset rules for decision optimization. Specifically, by presetting a reward function and state transition rules to guide the policy selection of an intelligent agent in a fixed scenario. This method has certain applicability in simple adversarial scenarios with strong determinacy, but exposes significant defects when facing complex dynamic adversarial environments. On the one hand, the static policy library is difficult to effectively represent the dynamic association relationships between environmental entities, and cannot capture the spatio-temporal evolution characteristics of adversarial logic in real time, resulting in limited policy adaptability. On the other hand, the decoupled design between the preset rules and the dynamic environment makes it difficult for the model to convert the complex relationships such as competition and restriction, and cooperation and support between environmental entities into quantifiable decision-making bases, resulting in a mismatch between policy selection and real-time situation.

[0003] Advanced reinforcement learning optimization techniques attempt to improve decision adaptability through a dynamic reward mechanism and environment modeling. These methods design dynamic optimization strategies, use environmental state transition constraint conditions, effectively integrate them into the reward function, and dynamically adjust the model structure to adapt to the changing adversarial environment. Although there is an improvement in feature expression ability, there are still fundamental limitations. First, the environmental modeling process is not deeply integrated with the structured knowledge system, and it is difficult to convert prior knowledge such as expert experience and adversarial logic into interpretable constraint conditions, resulting in the lack of rule guidance for policy generation. Second, the traditional dynamic adjustment mechanism fails to establish a two-way information interaction between knowledge reasoning and the policy network, and neither can optimize the model topology structure using the knowledge graph nor update the knowledge system through policy feedback. These problems seriously affect the practical value of the reinforcement learning model in real-time adversarial scenarios and limit the effectiveness of the intelligent decision-making system in dynamic game environments. Summary of the Invention

[0004] To solve the problems of low training efficiency of the reinforcement learning decision model and insufficient policy dynamic adaptability in an adversarial environment, the present invention provides an optimization method and system for a reinforcement learning model in an adversarial environment based on a knowledge graph.

[0005] In a first aspect, an optimization method for a reinforcement learning model in an adversarial environment based on a knowledge graph provided by the present invention adopts the following technical solution: An optimization method for a reinforcement learning model in an adversarial environment based on a knowledge graph includes: Obtain adversarial environment data and parse it into a state-action feature matrix, and construct a knowledge graph including environmental entities and adversarial logic; Design a dynamic optimization strategy for the reinforcement learning model based on the entity relationships in the knowledge graph, and generate the constraint conditions for the environmental state transition through graph reasoning; Encode the constraint conditions as a reward function and embed it into the reinforcement learning framework, and use the adversarial experience replay mechanism to dynamically adjust the model topology; Fuse the logical reasoning path of the knowledge graph and the reinforcement learning policy network to construct a two-way information interaction mechanism; Update the entity relationships in the knowledge graph according to the real-time adversarial data, generate a dynamic adversarial policy library, and improve the decision-making efficiency and accuracy of the model in complex adversarial environments.

[0006] Furthermore, the obtaining of adversarial environment data and parsing it into a state-action feature matrix, and constructing a knowledge graph including environmental entities and adversarial logic includes: Obtain adversarial environment data, perform multi-modal feature extraction on the original adversarial environment data, and separate the environmental state features and adversarial action features; Establish a set of adversarial logic relationships between entities, and the relationship types include competition and restriction, cooperation and support, and environmental impact; Construct a dimension alignment mechanism for the state-action feature matrix to map the discretized feature values to a unified dimension space; Use the graph embedding algorithm to encode the environmental entities and adversarial logic relationships as the nodes and edges of the knowledge graph, thus completing the construction of the knowledge graph including environmental entities and adversarial logic.

[0007] Furthermore, the dynamic optimization strategy for the reinforcement learning model based on the entity relationships in the knowledge graph includes: Extract the entity relationship weight matrix from the knowledge graph and establish a mapping function between the relationship strength and the policy selection probability; Generate a policy selection probability distribution based on the relationship weight matrix, and dynamically adjust the policy network parameters through the attention mechanism.

[0008] Furthermore, the generating of the constraint conditions for the environmental state transition through graph reasoning includes: Use the graph neural network for multi-hop reasoning to calculate the potential transfer probability between environmental entities; Combine the Dijkstra path search algorithm to generate the optimal state transition path; Generate a set of constraint conditions according to the entity relationship types in the path, including forbidden transfer conditions, mandatory transfer conditions, and transfer cost thresholds.

[0009] Furthermore, the encoding of the constraint conditions as a reward function and embedding it into the reinforcement learning framework includes: Construct a piecewise reward function according to the constraint conditions, and assign different reward values according to the satisfaction degree of the constraint conditions; Design a logic rule encoder to convert the constraint conditions generated by graph reasoning into differentiable mathematical expressions; Adopt the Lagrange multiplier method to dynamically adjust the constraint weight coefficient λ to achieve the adaptive fusion of constraint conditions and the original reward function.

[0010] Furthermore, the dynamic adjustment of the model topology structure by using the adversarial experience replay mechanism includes: Construct an adversarial sample filter to calculate the sample adversarial score according to the relationship strength of the knowledge graph; Construct a priority adversarial experience replay buffer for storing the historical experience data of the agent in the adversarial environment, including information such as state, reward, and constraint conditions; Adopt a neural network architecture search algorithm to dynamically adjust the network layer width and connection density according to the constraint state distribution of the samples in the buffer.

[0011] Furthermore, the construction of a two-way information interaction mechanism by integrating the logical reasoning path of the knowledge graph and the reinforcement learning policy network includes: Design a two-way attention mechanism in the embedding layer of the policy network to calculate the correlation weight between the knowledge graph path and the state features; Encode the logical reasoning path of the knowledge graph into the prior probability distribution of the policy network, and perform weighted fusion with the action probability output by the policy network; Establish a backpropagation channel to feedback the decision result of the policy network to the knowledge graph and update the relationship weight of the corresponding path.

[0012] Furthermore, the generation of a dynamic adversarial policy library by updating the entity relationship of the knowledge graph according to real-time adversarial data includes: Adopt an incremental graph update algorithm to dynamically adjust the entity relationship weight according to the newly collected adversarial data; Design a policy evaluation module to calculate the fitness score of the historical policy in the updated graph environment; Construct a hierarchical policy storage structure, and use the locality-sensitive hashing algorithm to establish a policy feature index to achieve fast retrieval and combination of policies.

[0013] In the second aspect, a reinforcement learning model optimization system based on a knowledge graph includes: A data acquisition module for acquiring adversarial environment data; A knowledge graph construction module, connected to the data acquisition module, for encoding environmental entities and adversarial logical relationships into nodes and edges of the knowledge graph by using a graph embedding algorithm, and constructing a knowledge graph containing environmental entities and adversarial logic; A reinforcement learning model dynamic optimization strategy design module, connected to the knowledge graph construction module, for designing a dynamic optimization strategy of the reinforcement learning model based on the entity relationship of the knowledge graph; A constraint condition generation module, connected to the reinforcement learning model dynamic optimization strategy design module and the knowledge graph construction module, is used to generate constraint conditions for the environmental state transition through graph reasoning; A reward function fusion module, connected to the constraint condition generation module, is used to encode the constraint conditions as a reward function and embed it into the reinforcement learning framework; A topology structure adjustment module, connected to the reward function fusion module, is used to dynamically adjust the model topology structure by using the adversarial experience replay mechanism; A bidirectional interaction module, connected to the knowledge graph construction module and the topology structure adjustment module, is used to fuse the logical reasoning path of the knowledge graph and the reinforcement learning policy network to construct a bidirectional information interaction mechanism; A dynamic policy library management module, connected to the bidirectional interaction module and the knowledge graph construction module, is used to update the entity relationship of the knowledge graph according to real-time adversarial data and generate a dynamic adversarial policy library.

[0014] Thirdly, the present invention provides a computer-readable storage medium, in which multiple instructions are stored, and the instructions are suitable for being loaded and executed by a processor of a terminal device for the method for constructing a knowledge graph related to the situation and action of marine equipment as described above.

[0015] Fourthly, the present invention provides a terminal device, including a processor and a computer-readable storage medium. The processor is used to implement each instruction; the computer-readable storage medium is used to store multiple instructions, and the instructions are suitable for being loaded and executed by the processor for the method for constructing a knowledge graph related to the situation and action of marine equipment as described above.

[0016] In summary, the present invention has the following beneficial technical effects: 1. The optimization method and system for the adversarial environment reinforcement learning model based on the knowledge graph proposed by the present invention deeply integrates the logical reasoning ability of the knowledge graph and the policy optimization mechanism of reinforcement learning to construct a bidirectional dynamic interaction framework. Compared with the traditional reinforcement learning model, this method effectively solves the state-action space explosion problem in complex adversarial environments by using the structured reasoning ability of the knowledge graph. Through constraint condition encoding and the adversarial experience replay mechanism, the training efficiency of the reinforcement learning model is effectively improved, and the adaptability of the dynamic policy network is enhanced.

[0017] 2. The multi-modal feature extraction and state-action feature matrix dimension alignment mechanism adopted by the present invention maps the discretized adversarial features to a unified dimension space through a graph embedding algorithm, solving the problems of mismatched dimensions between environmental state features and adversarial action features and incomplete representation of logical relationships in traditional methods. The constructed knowledge graph can simultaneously encode three core adversarial logics of competition and restriction, cooperation and support, and environmental impact, greatly enhancing the representation accuracy of adversarial relationships, which is much higher than the traditional single-modal modeling method.

[0018] 3. The proposed method of fusing bidirectional attention mechanism and policy network in the present invention encodes the inference path of the knowledge graph into the prior probability distribution of the policy network and establishes a backpropagation channel to achieve dynamic weight adjustment, effectively solving the problem of the disconnection between logical reasoning and policy decision-making in traditional methods. This method significantly improves the accuracy of policy selection in complex adversarial scenarios, while significantly reducing ineffective exploration actions and greatly enhancing the intelligent level of adversarial decision-making.

[0019] 4. The designed incremental graph update algorithm and hierarchical policy storage architecture in the present invention achieve the co-evolution of the knowledge graph and the policy library through dynamically adjusting entity relationship weights and local sensitive hashing indexing technology. Compared with the static policy library method, this mechanism greatly improves the adaptation speed of the model to the new adversarial environment, significantly improves the policy retrieval efficiency, and can ensure that the policy combination maintains a high success rate level, guaranteeing the continuous optimization ability of the system in the dynamic adversarial environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 is the flowchart of the method in Embodiment 1 of the present invention; Figure 2 is the schematic structural diagram of the system in Embodiment 2 of the present invention; DETAILED DESCRIPTION OF THE INVENTION

[0021] The present invention will be further described in detail below with reference to the accompanying drawings.

[0022] Embodiment 1 Referring to Figure 1 , a method for optimizing an adversarial environment reinforcement learning model based on a knowledge graph in this embodiment includes: S1. Obtain adversarial environment data and parse it into a state-action feature matrix, and construct a knowledge graph including environmental entities and adversarial logic, including: S11. Obtain adversarial environment data, perform multi-modal feature extraction on the original adversarial environment data, and separate environmental state features and adversarial action features; Collect original adversarial environment data from the simulation system, process the original data stream through a spatio-temporal feature decoupling network. For environmental state feature extraction, 3D convolution is used to capture the spatial distribution pattern, and for adversarial action feature extraction, a temporal autoencoder is used to capture the action sequence law. The two types of features are orthogonally constrained in the latent space to ensure the independence of feature decoupling. Through this multi-modal feature extraction method, the effective separation of static attributes and dynamic behaviors in the adversarial environment is achieved, laying a foundation for subsequent analysis.

[0023] S12. Establish a set of adversarial logic relationships between entities, and the relationship types include three types: competition and restriction, cooperation and support, and environmental impact; Define the rules for determining the type of relationship: When there is a negative correlation in resource consumption between entities, it is marked as a competitive restriction relationship; when there is a possibility of tactical cooperation, it is marked as a collaborative support relationship; when the entity attributes are affected by environmental parameters exceeding a threshold, an environmental impact relationship is established. Analyze the extracted environmental entities and adversarial actions to determine the logical relationships between entities, and formalize the adversarial logic through a relationship type matrix.

[0024] S13. Construct a dimension alignment mechanism for the state-action feature matrix to map the discretized eigenvalue to a unified dimension space; For different types of features, different discretization methods are adopted. For continuous environmental state features, they are divided into several intervals according to their value ranges for discretization; for discrete adversarial action features, direct numerical encoding is performed. Then, through a normalization method, these discretized eigenvalues are mapped to the same dimension space so that they can be uniformly represented and processed in the same matrix.

[0025] S14. Use a graph embedding algorithm to encode environmental entities and adversarial logical relationships as nodes and edges of a knowledge graph, thus completing the construction of a knowledge graph containing environmental entities and adversarial logic; Use a graph attention network for node embedding, and a relational graph convolutional network for edge feature encoding. Aggregate neighborhood information through a multi-hop message passing mechanism, and finally generate a three-dimensional knowledge graph containing environmental entity nodes, adversarial logic edges, and relationship strength weights, providing a structured data basis for subsequent graph-based analysis and model optimization.

[0026] S2. Design a dynamic optimization strategy for the reinforcement learning model based on the entity relationships in the knowledge graph. The constraint conditions for environmental state transition generated through graph reasoning include: S21. Extract the entity relationship weight matrix from the knowledge graph and establish a mapping function between the relationship strength and the policy selection probability; Construct a relationship strength quantization model, and statistically calculate the interaction frequency and success probability between entities based on historical adversarial data. Use the sigmoid function to map the relationship strength to the interval [0,1] to form a relationship weight matrix. Based on this weight matrix, establish a linear mapping function to map the relationship strength to the policy selection probability. The greater the relationship strength, the higher the probability of the corresponding policy being selected, enabling the model to give priority to strongly associated policies when making decisions.

[0027] S22. Generate a policy selection probability distribution based on the relationship weight matrix and dynamically adjust the policy network parameters through an attention mechanism; The Softmax function is used to convert the relationship weight matrix into a policy selection probability distribution, and a selection probability is assigned to each policy. An attention mechanism is introduced into the policy network to dynamically adjust the parameter weights of different parts of the policy network according to the current state and the relationship weights in the knowledge graph. This can enhance the learning and execution capabilities of the corresponding policies, thereby optimizing the decision-making performance of the policy network.

[0028] S23. Use a graph neural network for multi-hop reasoning to calculate the potential transition probabilities between environmental entities Use the GraphSAGE graph neural network algorithm to perform multi-hop reasoning on the knowledge graph. Starting from the current environmental entity node, traverse the neighbor nodes through multiple hops, and combine the node attributes and the relationships of the edges to calculate the potential transition probabilities between different entities.

[0029] S24. Combine the Dijkstra path search algorithm to generate the optimal state transition path; Convert the knowledge graph into a weighted directed graph with the calculated potential transition probabilities as weights. Use the Dijkstra path search algorithm to find the optimal path from the current state to the target state in the knowledge graph. Find a state transition path that can meet the target requirements and minimize the overall cost.

[0030] S25. Generate a set of constraint conditions according to the entity relationship types in the path, including prohibited transfer conditions, mandatory transfer conditions, and transfer cost thresholds; Analyze the relationship types between nodes in the optimal state transition path to generate corresponding constraint conditions. For example, a competitive constraint relationship can generate a prohibited transfer condition; a collaborative support relationship can generate a mandatory transfer condition; an environmental impact relationship can generate a condition that requires careful decision-making or avoids transfer.

[0031] S3. Encode the constraint conditions as a reward function and embed it into the reinforcement learning framework, and use the adversarial experience replay mechanism to dynamically adjust the model topology, including: S31. Construct a piecewise reward function according to the constraint conditions, and assign different reward values according to the degree of satisfaction of the constraint conditions; Classify the constraint conditions into different priorities and types, and construct a piecewise reward function. For decisions that meet the mandatory transfer conditions, give a higher positive reward; for decisions that violate the prohibited transfer conditions, give a larger negative reward; for decisions within the transfer cost threshold range, give different degrees of rewards according to the degree of closeness to the optimal value.

[0032] S32. Design a logical rule encoder to convert the constraint conditions generated by graph reasoning into differentiable mathematical expressions; The fuzzy logic system is adopted to convert symbolic constraints into continuously differentiable expressions, so as to enable gradient calculation and optimization in the reinforcement learning framework. An adaptive membership function is designed to adjust the strictness of the constraints through learnable parameters. The encoder output is concatenated with the original reward function in a tensor.

[0033] S33. The Lagrange multiplier method is used to dynamically adjust the constraint weight coefficient λ to achieve the adaptive fusion of the constraint conditions and the original reward function; The Lagrange multiplier method is introduced to combine the constraint conditions with the original reward function. By dynamically adjusting the constraint weight coefficient λ, the relationship between the constraint conditions and the original reward is balanced. In the initial stage of training, in order to quickly explore feasible strategies, the constraint weight can be appropriately reduced; as the training progresses, the constraint weight is gradually increased to ensure that the model optimizes the reward on the premise of meeting the constraint conditions. According to the performance of the model during training and the satisfaction of the constraint conditions, the value of λ is dynamically adjusted to achieve the adaptive fusion of the constraints and the rewards.

[0034] S34. An adversarial sample filter is constructed to calculate the adversarial score of the sample according to the relationship strength in the knowledge graph; During the adversarial experience replay process, an adversarial sample filter is constructed. According to the relationship strength between entities in the knowledge graph, the adversarial score of each sample is calculated. Samples containing strong competitive restrictive relationships or complex environmental impact relationships are given a higher adversarial score; conversely, a lower score is given. In this way, adversarial samples with higher value are screened out for model training to improve the training efficiency and the adversarial ability of the model.

[0035] S35. A priority adversarial experience replay buffer is constructed to store the historical experience data of the agent in the adversarial environment, including information such as states, rewards, and constraint conditions; A priority adversarial experience replay buffer is designed to store the experience data in queues with different priorities according to the adversarial scores of the samples. High-score samples are stored in the high-priority queue and are preferentially taken out for model training; low-score samples are stored in the low-priority queue and participate in training at an appropriate time. This can ensure that the model preferentially learns the most challenging and valuable experiences, accelerating the convergence and optimization of the model.

[0036] S36. The neural network architecture search algorithm is adopted to dynamically adjust the width of the network layer and the connection density according to the constraint state distribution of the samples in the buffer; The neural architecture search algorithm is adopted to automatically adjust the width of the network layer according to the constraint satisfaction rate. A connection density controller is designed to adjust the number of network connections based on the change rate of the policy entropy value. A dynamic pruning mechanism is introduced to remove network connections with low contribution to the current constraint state.

[0037] S4. Integrate the logical reasoning path of the knowledge graph and the reinforcement learning policy network to construct a two-way information interaction mechanism, including: S41. Design a two-way attention mechanism in the policy network embedding layer to calculate the correlation weights between the knowledge graph path and the state features; Introduce a two-way attention mechanism in the embedding layer of the policy network to calculate the correlation weights in two directions, namely from the state features to the knowledge graph path and from the knowledge graph path to the state features. Use the multi-head attention mechanism to process the interaction patterns of different relationship types in parallel. The attention weights are used as prior knowledge to guide feature fusion.

[0038] S42. Encode the logical reasoning path of the knowledge graph as the prior probability distribution of the policy network, and perform weighted fusion with the action probability output by the policy network; Encode the optimal path or key path obtained through logical reasoning in the knowledge graph as the prior probability distribution of the policy network. This prior probability distribution reflects the policy tendency obtained based on the experience and reasoning of the knowledge graph. Perform weighted fusion with the action probability learned by the policy network through the current state. When making decisions, consider both the real-time information of the current state and the prior knowledge in the knowledge graph to improve the accuracy and stability of decisions.

[0039] S43. Establish a backpropagation channel to feedback the decision result of the policy network to the knowledge graph and update the relationship weights of the corresponding paths; Establish a backpropagation channel between the policy network and the knowledge graph, design a contrastive loss function, and adjust the edge weights of the knowledge graph through gradient backpropagation. If the decision brings high rewards, enhance the relationship weights of the relevant paths; if the decision leads to low rewards or failures, reduce the relationship weights of the relevant paths, so that the knowledge graph can continuously evolve and optimize according to the actual confrontation experience.

[0040] S5. Update the entity relationships of the knowledge graph according to the real-time confrontation data to generate a dynamic confrontation policy library, including: S51. Adopt an incremental graph update algorithm to dynamically adjust the entity relationship weights according to the newly collected confrontation data; As the real-time confrontation data is continuously collected, use the incremental graph update algorithm to update the knowledge graph. When new entities or relationships appear in the new data, add them to the knowledge graph; for the existing entity relationships, dynamically adjust their weights according to the information in the new data.

[0041] S52. Design a policy evaluation module to calculate the fitness score of historical policies in the updated graph environment; Build a strategy evaluation module, adopt the transfer test method, and simulate the execution of historical strategies in the updated knowledge graph environment. Design a multi-dimensional fitness evaluation function, including three core indicators: strategy success rate, resource efficiency, and confrontation effect, to comprehensively evaluate the quality of strategies.

[0042] S53. Build a hierarchical strategy storage structure, adopt the locality-sensitive hashing algorithm to establish a strategy feature index, and realize the fast retrieval and combination of strategies; Adopt a four-layer storage architecture: the basic layer stores atomic action strategies, the combination layer stores tactical combinations, the scenario layer stores environment adaptation strategies, and the meta layer stores strategy generation rules. Design a locality-sensitive hashing index to achieve fast approximate neighbor retrieval through strategy feature vectors. When it is necessary to quickly retrieve and combine strategies in dynamic confrontation, relevant strategies can be quickly located through the feature index, improving the efficiency of strategy generation and application, and meeting the real-time decision-making needs in complex confrontation environments.

[0043] This embodiment provides a method and system for optimizing a reinforcement learning model in an adversarial environment based on a knowledge graph. This method and system are proposed in the context of low training efficiency of the reinforcement learning model and difficult adaptation of strategies to dynamic changes in the current adversarial environment. In complex adversarial scenarios, traditional models are difficult to effectively process massive data and complex logic, resulting in insufficient decision-making capabilities. In order to achieve knowledge-driven intelligent decision optimization in massive adversarial environment data, feature alignment and dimension mapping of multi-source heterogeneous data are performed through the state-action feature matrix analysis method, and a dynamic knowledge graph integrating environmental entity nodes and adversarial logical relationships is constructed. First, the multi-modal feature decoupling technology is used to separate the environmental state features and adversarial action features, and an adversarial logical relationship set including three-dimensional relationships of competition and restriction, cooperation and support, and environmental impact is established; then, a set of constraint conditions for environmental state migration is generated through multi-hop reasoning of the graph neural network and the improved Dijkstra algorithm, and a differentiable logic encoder is designed to convert symbolic constraints into a reward function in the reinforcement learning framework; then, the adversarial experience replay mechanism is combined with the dynamic network topology adjustment technology to construct a two-way attention interaction channel between the knowledge reasoning path and the policy network; finally, the real-time evolution of the knowledge graph is realized through the incremental graph update algorithm, and a hierarchical dynamic adversarial strategy library is generated. This method improves the training efficiency and strategy dynamic adaptability of the reinforcement learning model in the adversarial environment, provides strong support for accurate and real-time intelligent decision-making on the massive real-time situation data of a large number of combat units and a large number of combat rules in actual scenarios, and improves the accuracy and timeliness of decision-making in complex adversarial scenarios.

[0044] Embodiment 2 Refer to Figure 2 , this embodiment provides a system for optimizing a reinforcement learning model in an adversarial environment based on a knowledge graph, including: A data acquisition module for acquiring adversarial environment data; A knowledge graph construction module, connected to the data acquisition module, for encoding environmental entities and adversarial logical relationships as nodes and edges of a knowledge graph using a graph embedding algorithm, and constructing a knowledge graph containing environmental entities and adversarial logic; A reinforcement learning model dynamic optimization strategy design module, connected to the knowledge graph construction module, for designing a dynamic optimization strategy of the reinforcement learning model based on the entity relationships of the knowledge graph; A constraint condition generation module, connected to the reinforcement learning model dynamic optimization strategy design module and the knowledge graph construction module, for generating constraint conditions for environmental state transition through graph reasoning; A reward function fusion module, connected to the constraint condition generation module, for encoding the constraint conditions as a reward function and embedding it into the reinforcement learning framework; A topology structure adjustment module, connected to the reward function fusion module, for dynamically adjusting the model topology structure using an adversarial experience replay mechanism; A two-way interaction module, connected to the knowledge graph construction module and the topology structure adjustment module, for fusing the logical reasoning path of the knowledge graph and the reinforcement learning policy network to construct a two-way information interaction mechanism; A dynamic policy library management module, connected to the two-way interaction module and the knowledge graph construction module, for updating the entity relationships of the knowledge graph according to real-time adversarial data and generating a dynamic adversarial policy library.

[0045] A computer-readable storage medium storing multiple instructions, which are adapted to be loaded and executed by a processor of a terminal device for the method for constructing a knowledge graph related to the situation and action of a maritime equipment as described above.

[0046] A terminal device, including a processor and a computer-readable storage medium, where the processor is used to implement each instruction; the computer-readable storage medium is used to store multiple instructions, which are adapted to be loaded and executed by the processor for the method for constructing a knowledge graph related to the situation and action of a maritime equipment as described above.

[0047] The above are all preferred embodiments of the present invention, and the protection scope of the present invention is not limited thereby. Therefore, all equivalent changes made according to the structure, shape, and principle of the present invention should be covered within the protection scope of the present invention.

Claims

1. A knowledge graph-based adversarial environment reinforcement learning model optimization method, characterized in that: include: Obtain adversarial environment data and parse it into a state-action feature matrix to build a knowledge graph containing environment entities and adversarial logic; Design dynamic optimization strategies for reinforcement learning models based on entity relationships in knowledge graphs, and generate constraints for environmental state migration through graph reasoning; The constraints are encoded as reward functions and embedded into the reinforcement learning framework, and the model topology is dynamically adjusted using the adversarial experience replay mechanism. Integrate the logical reasoning path of the knowledge graph with the reinforcement learning strategy network to build a two-way information interaction mechanism; Update the knowledge graph entity relationships based on real-time adversarial data, generate a dynamic adversarial strategy library, and improve the decision-making efficiency and accuracy of the model in complex adversarial environments.

2. According to the knowledge graph-based adversarial environment reinforcement learning model optimization method of claim 1, it is characterized in that: The acquisition of adversarial environment data and parsing into a state-action feature matrix, and construction of a knowledge graph containing environment entities and adversarial logic include: Acquire adversarial environment data, perform multimodal feature extraction on the original adversarial environment data, and separate environmental state features and adversarial action features; Establishing a set of confrontational logical relationships between entities, wherein the relationship types include competitive constraints, collaborative support, and environmental influences; Construct a dimension alignment mechanism for the state-action feature matrix to map the discretized feature values ​​to a unified dimensional space; A graph embedding algorithm is used to encode the environmental entities and adversarial logic relationships into nodes and edges of the knowledge graph, thereby completing the construction of a knowledge graph containing environmental entities and adversarial logic.

3. According to the knowledge graph-based adversarial environment reinforcement learning model optimization method of claim 1, it is characterized in that: The dynamic optimization strategy of the entity relationship design reinforcement learning model based on the knowledge graph includes: Extract the entity relationship weight matrix from the knowledge graph and establish a mapping function between relationship strength and strategy selection probability; The strategy selection probability distribution is generated based on the relationship weight matrix, and the strategy network parameters are dynamically adjusted through the attention mechanism.

4. According to the knowledge graph-based adversarial environment reinforcement learning model optimization method of claim 1, it is characterized in that: The constraints for generating the environment state migration through graph reasoning include: Use graph neural networks for multi-hop reasoning and calculate potential transition probabilities between environmental entities; Combine Dijkstra path search algorithm to generate the optimal state migration path; A set of constraint conditions is generated according to the entity relationship type in the path, including prohibited transfer conditions, required transfer conditions and transfer cost thresholds.

5. According to the knowledge graph-based adversarial environment reinforcement learning model optimization method of claim 1, it is characterized in that: The encoding of constraints into a reward function and embedding it into a reinforcement learning framework includes: Construct a piecewise reward function based on the constraints, and assign different reward values ​​according to the degree of satisfaction of the constraints; Design a logic rule encoder to convert the constraints generated by graph reasoning into differentiable mathematical expressions; The Lagrange multiplier method is used to dynamically adjust the constraint weight coefficient λ to achieve adaptive fusion of the constraint conditions and the original reward function.

6. According to the knowledge graph-based adversarial environment reinforcement learning model optimization method of claim 1, it is characterized in that: The dynamically adjusting the model topology structure by using the adversarial experience replay mechanism includes: Construct an adversarial sample filter to calculate the sample adversarial score based on the strength of the knowledge graph relationship; Construct a priority adversarial experience replay buffer to store the agent's historical experience data in the adversarial environment, including information such as status, rewards, and constraints; A neural network architecture search algorithm is used to dynamically adjust the network layer width and connection density according to the constraint state distribution of samples in the buffer.

7. According to the knowledge graph-based adversarial environment reinforcement learning model optimization method of claim 1, it is characterized in that: The logical reasoning path of the fusion knowledge graph and the reinforcement learning strategy network to build a two-way information interaction mechanism includes: Design a bidirectional attention mechanism in the policy network embedding layer to calculate the association weights between the knowledge graph path and state features; Encode the logical reasoning path of the knowledge graph into the prior probability distribution of the policy network and perform weighted fusion with the action probability output by the policy network; Establish a back-propagation channel, feed back the decision results of the policy network to the knowledge graph, and update the relationship weights of the corresponding paths.

8. According to the knowledge graph-based adversarial environment reinforcement learning model optimization method of claim 1, it is characterized in that: The updating of the knowledge graph entity relationship according to the real-time confrontation data and the generation of a dynamic confrontation strategy library include: Adopting an incremental graph update algorithm, the entity relationship weights are dynamically adjusted according to the newly collected adversarial data; Design a strategy evaluation module to calculate the fitness score of the historical strategy in the updated graph environment; Build a hierarchical strategy storage structure and use the local sensitive hashing algorithm to establish a strategy feature index to achieve fast retrieval and combination of strategies.

9. A knowledge graph-based adversarial environment reinforcement learning model optimization system, comprising: A data acquisition module, used to acquire adversarial environment data; The knowledge graph construction module is connected to the data acquisition module and is used to encode the relationship between the environmental entities and the adversarial logic into nodes and edges of the knowledge graph using a graph embedding algorithm, thereby constructing a knowledge graph containing the environmental entities and the adversarial logic; The reinforcement learning model dynamic optimization strategy design module is connected to the knowledge graph construction module and is used to design the dynamic optimization strategy of the reinforcement learning model based on the entity relationship of the knowledge graph; The constraint generation module is connected with the reinforcement learning model dynamic optimization strategy design module and the knowledge graph construction module to generate constraint conditions for environment state migration through graph reasoning; The reward function fusion module is connected to the constraint generation module and is used to encode the constraints into a reward function and embed it into the reinforcement learning framework; The topology adjustment module is connected to the reward function fusion module and is used to dynamically adjust the model topology using the adversarial experience replay mechanism; The two-way interaction module is connected with the knowledge graph construction module and the topology structure adjustment module to integrate the logical reasoning path of the knowledge graph with the reinforcement learning strategy network to build a two-way information interaction mechanism; The dynamic strategy library management module is connected with the two-way interaction module and the knowledge graph construction module, and is used to update the knowledge graph entity relationship according to the real-time adversarial data and generate a dynamic adversarial strategy library.

Images