A graph layering reinforcement learning method based on edge weight modeling, terminal and storage medium

Abstract

The application relates to the technical field of layered reinforcement learning, and discloses a graph layered reinforcement learning method based on edge weight modeling, a terminal and a storage medium. The method randomly samples a first candidate node set from trajectory data of a lower-layer buffer pool, randomly samples a second candidate node set from trajectory data of an upper-layer buffer pool, and selects a plurality of target nodes from the candidate node sets to form a graph structure node set; for any two nodes in the graph structure node set, state transition edge weights, energy distance edge weights and neural network prediction edge weights are respectively calculated, the three kinds of edge weights are weighted and fused to calculate total edge weight values; when the total edge weight values between the two nodes are not less than a preset threshold, the corresponding edges are retained and the total edge weight values are added to the edges to form a graph structure with semantic edge weights; and the graph structure is applied to strategy training and path planning of layered reinforcement learning. The application improves the representativeness and globality of the graph structure, and the graph structure has stronger expression capability and interpretability.

Description

Technical Field

[0001] This invention relates to the field of hierarchical reinforcement learning technology, specifically to a graph hierarchical reinforcement learning method, terminal, and storage medium based on edge weight modeling. Background Technology

[0002] Hierarchical Reinforcement Learning (HRL) is an important method for solving complex tasks. It decomposes and solves tasks step-by-step through the division of labor between high-level and low-level policies. In recent years, researchers have introduced graph structures to enhance the expressive power of HRL, modeling states, actions, or subtasks in the environment as nodes and edges of a graph, enabling agents to make decisions at different levels. In this way, graph structures can not only represent dependencies and transition rules between tasks but also provide a clearer view of task decomposition for complex tasks, thereby improving the ability to handle long-term dependent tasks. Currently, the main applications of graph-based hierarchical reinforcement learning are path planning and robot training, essentially training an agent to learn how to complete a task.

[0003] In existing research, graph-layered reinforcement learning typically involves uniformly sampling a batch of trajectory data as candidate nodes in a lower-level buffer pool, and then selecting a certain number of nodes from these as components of the graph. Subsequently, the task graph is constructed by adding edges to each node. However, this method mainly relies on local random sampling, which can easily lead to the constructed graph being concentrated in local high-frequency regions, thus ignoring the global task structure and making it difficult to fully reflect the key states and sub-objective boundaries in the task.

[0004] Furthermore, existing methods focus primarily on node selection during graph construction, while neglecting semantic modeling of edges. In most current graph structures, edges are treated merely as "unweighted edges" or "fixed connections," failing to reflect reachability, cost, or semantic relevance between different states. This limits the graph's expressive power, making it difficult to fully utilize state transition characteristics during policy training, thus impacting the efficiency of path planning and global task decision-making.

[0005] In summary, existing graph hierarchical reinforcement learning methods still have shortcomings in terms of the globality of node sampling and the semantic modeling of edge weights. They are unable to take into account both the global task structure and local state features, thus affecting the learning efficiency and policy quality of agents in complex tasks. Summary of the Invention

[0006] To address the technical problems of existing graph hierarchical reinforcement learning methods, such as node sampling being easily limited to local regions and insufficient expressive power due to the lack of semantic modeling of edges in the graph, this invention provides a graph hierarchical reinforcement learning method, terminal, and storage medium based on edge weight modeling.

[0007] To achieve the above objectives, the present invention provides the following technical solution:

[0008] This invention discloses a graph hierarchical reinforcement learning method based on edge weight modeling, which includes the following steps, namely S1~S4.

[0009] S1. Randomly sample the trajectory data from the lower buffer pool to obtain the first candidate node set, and randomly sample the trajectory data from the upper buffer pool to obtain the second candidate node set.

[0010] S2. Select several target nodes from the first candidate node set and the second candidate node set to form a graph structure node set.

[0011] S3. For any two nodes in the graph structure node set, calculate the state transition edge weight, energy distance edge weight, and neural network prediction edge weight respectively, and then weight and fuse the three edge weights to calculate the total edge weight value; wherein, the state transition edge weight is determined based on the reachability probability from one node state to another node state under the current policy; the energy distance edge weight is determined based on the Euclidean distance between the state spaces of the nodes and the energy of the required action; the neural network prediction edge weight is determined based on the trained scoring model, which takes the triple formed by the node state pair and the ideal target as input, and performs nonlinear mapping and feature fusion through a multilayer perceptron structure to generate a semantic score value representing the reachability, task relevance, and execution success probability between the node state pairs.

[0012] S4. When the total edge weight between two nodes is not less than a preset threshold, the corresponding edge is retained and the total edge weight is added to the edge to form a graph structure with semantic edge weights, thereby applying the graph structure to policy training and path planning in hierarchical reinforcement learning.

[0013] As a further improvement to the above scheme, in step S2, the target node is selected based on the preset weighted farthest point sampling algorithm. The specific process is as follows, namely S21~S23.

[0014] S21. Use an array to store the result set, and set the node state weights in the first and second candidate node sets:

[0015] ;

[0016] In the formula, For nodes i state, Indicates node state The weights; Represents the first candidate node set; Indicates the second candidate node set; >1.

[0017] S22. Randomly select a node from the first candidate node set and the second candidate node set. i As the initial node, the weight distribution of the states of all candidate nodes is calculated each time the next node is selected, expressed by the following formula:

[0018] ;

[0019] In the formula, N express The set of all nodes except the selected node. j For set N Any node in; Represents a set N State of each node The weight distribution; Softmax[·] is the normalized exponential function; Indicates node state With node status The distance between them; Indicates node state The weight.

[0020] S23. Select the node with the largest weight from the weight distribution and add it to the result set until the number of arrays reaches the total number of nodes in the graph structure node set.

[0021] As a further improvement to the above scheme, in step S22, the node state... With node status The formula for calculating the distance between them is:

[0022] ;

[0023] In the formula, To enhance the discount in learning, Indicates node state After taking the action, the node state is reached. The expected return is output through the value network; Indicates the state of the slave node. To the node state During the process, the action of policy selection is output by the policy network.

[0024] As a further improvement to the above scheme, in step S3, a node is set under the current strategy. i status As a starting point and another node j status As a sub-objective, multi-step execution is simulated to compute nodes. i and nodes jState transition edge weights between The weight of the state transition edge is positively correlated with the level of reachability, expressed by the formula:

[0025] ;

[0026] In the formula, Indicates that no target is specified. In this case, the lower-level intelligent agent executes t At each time step, it can eventually reach... The number of simulations; Indicates that the specified target is Under these circumstances, it will eventually be possible to reach The number of simulations.

[0027] As a further improvement to the above scheme, in step S3, the expression formula for the energy transfer edge weight is:

[0028] ;

[0029] In the formula, Indicates node state and node status The Euclidean distance between them; Indicates the state of the slave node to state The energy required for the action; To adjust the hyperparameters for state difference and the proportion of motion energy; Represents the L2 norm; Indicates node state and node status The energy distance between the edges.

[0030] As a further improvement to the above scheme, in step S3, the samples used to train the scoring model include positive samples and negative samples; wherein, positive samples are continuous state pairs extracted from actual trajectory data, corresponding to feasible path points where the strategy has been executed; negative samples are discontinuous state pairs extracted from failed trajectories, and binary classification labels are constructed. , is represented as:

[0031] .

[0032] As a further improvement to the above scheme, the following loss function is used when training the scoring model:

[0033] ;

[0034] In the formula, Indicates loss; This indicates that the scoring model is for triples. The edge weight prediction is the normalized result of the semantic score output by the multilayer perceptron structure; where... dg t Indicates an ideal goal;

[0035] The formula for expressing the edge weights predicted by the neural network is as follows:

[0036] ;

[0037] In the formula, Represents a node i and nodes j Neural network prediction of edge weights between them; This represents the scoring process of a multilayer perceptron network structure. This indicates normalization processing;

[0038] The formula for calculating the total edge weight is as follows:

[0039] ;

[0040] In the formula, α , β and γ These are the weights corresponding to the state transition edge weight, energy distance edge weight, and neural network prediction edge weight, respectively.

[0041] As a further improvement to the above scheme, in step S1, the number of trajectories in the lower buffer pool and the upper buffer pool is the same; each trajectory in the lower buffer pool contains T1 time steps, and the format of each time step is as follows: Several time steps are sampled from the lower buffer pool as the first candidate node set; each trajectory in the upper buffer pool contains T2 time steps, and the format of each time step is as follows: Where T1 > T2, Indicates the node status. This indicates the goal achieved by the lower-level intelligent agent. Indicates sub-goals, Indicates the action to be performed. It represents an ideal goal.

[0042] The present invention also discloses a computer terminal, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the graph hierarchical reinforcement learning method based on edge weight modeling as described above.

[0043] The present invention also discloses a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the graph hierarchical reinforcement learning method based on edge weight modeling as described above.

[0044] Compared with the prior art, the beneficial effects of the present invention are:

[0045] This invention discloses a graph-based hierarchical reinforcement learning method based on edge weight modeling. By introducing a hybrid sampling mechanism of upper and lower layer buffer pools, it can prioritize the coverage of global key states and sub-objective boundaries in candidate nodes, avoiding the shortcomings of traditional methods that only focus on local high-frequency regions. This improves the representativeness and globality of the graph structure and more comprehensively reflects the hierarchical relationships of the task. Furthermore, this invention proposes a fusion mechanism for edge weight modeling. By combining state transition probability, energy distance, and neural network prediction, semantic weights are assigned to edges in the graph. This not only reflects the reachability and energy cost between nodes but also the task relevance and execution success rate. The constructed graph structure has stronger expressive power and interpretability, significantly improving the convergence speed of policy training, the rationality of path planning, and the overall stability of task execution in hierarchical reinforcement learning. Attached Figure Description

[0046] Figure 1 This is a flowchart of the graph hierarchical reinforcement learning method based on edge weight modeling in Embodiment 1 of the present invention.

[0047] Figure 2 This is a comparative experimental result diagram of various mapping methods in Embodiment 1 of the present invention.

[0048] Figure 3 This is a schematic diagram of the structure of the computer terminal in Embodiment 2 of the present invention. Detailed Implementation

[0049] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0050] Example 1

[0051] Please see Figure 1 This embodiment provides a graph hierarchical reinforcement learning method based on edge weight modeling, which includes the following steps, namely S1~S4.

[0052] S1. Randomly sample the trajectory data from the lower buffer pool to obtain the first candidate node set, and randomly sample the trajectory data from the upper buffer pool to obtain the second candidate node set.

[0053] In this embodiment, the lower buffer pool contains a total of 200 trajectories, each containing 600 time steps, and each time step is formatted as follows: The samples were obtained by random sampling from the lower buffer pool. n l Each time step is used as the first candidate node set. n l =4500

[0054] The upper buffer pool also contains 200 tracks, but each track only contains 15 time steps, and the format of each time step is as follows: Random sampling was used to obtain samples from the upper buffer pool. n h Each time step is used as the second candidate node set. n h =500

[0055] Indicates the node status. This indicates the goal achieved by the lower-level intelligent agent. Indicates sub-goals, Indicates the action to be performed. It represents an ideal goal.

[0056] This invention introduces a multi-buffered hybrid sampling mechanism. The upper-level policy buffer pool stores states related to sub-goal selection and global decision-making, possessing advantages in criticality and diversity. It prioritizes retaining critical states such as forks in a maze or grasping moments in robot operations, which typically represent important decision points in the task. Because the higher-level policy has a longer field of view and broader coverage, this buffer pool can form a global representation of the task structure, encompassing not only task bottlenecks and sub-goal boundaries but also avoiding repeated sampling of local states. Compared to traditional methods that rely solely on lower-level policies, this design more effectively constructs a graph structure reflecting the task hierarchy, improving the system's ability to balance local optimization and global exploration.

[0057] S2. Select from the first candidate node set and the second candidate node set. n Each target node forms a graph structure node set.

[0058] In step S2, the target node is selected based on the preset weighted farthest point sampling algorithm. The specific process is as follows, namely S21~S23.

[0059] S21. Using an array res [ n ], used to store the result set, and set the node state weights in the first candidate node set and the second candidate node set:

[0060] ;

[0061] In the formula, For nodes i state, Indicates node state The weights; Represents the first candidate node set; Indicates the second candidate node set; >1.

[0062] S22. Randomly select a node from the first candidate node set and the second candidate node set. i As the initial node, the weight distribution of the states of all candidate nodes is calculated each time the next node is selected, expressed by the following formula:

[0063] ;

[0064] In the formula, N express The set of all nodes except the selected node. j For set N Any node in; Represents a set N State of each node The weight distribution; Softmax[·] is the normalized exponential function; Indicates node state With node status The distance between them; Indicates node state The weight.

[0065] In step S22, the node state With node status The formula for calculating the distance between them is:

[0066] ;

[0067] In the formula, To enhance the discount in learning, Indicates node state After taking the action, the node state is reached. The expected return is output through the value network; Indicates the state of the slave node. To the node state During the process, the action of policy selection is output by the policy network.

[0068] S23. Select the node with the largest weight from the weight distribution and add it to the result set until the array size reaches a certain threshold. n That is, the total number of nodes in the graph structure node set.

[0069] S3. For any two nodes in the graph structure node set, calculate the state transition edge weight, energy distance edge weight, and neural network prediction edge weight respectively, and then weight and fuse the three edge weights to calculate the total edge weight.

[0070] In this embodiment, it is necessary to provide n Each candidate node is given an edge, and a weight is calculated for each edge, thereby accurately modeling the connection relationship between any two nodes and improving the graph's expressiveness, reasoning ability, and policy guidance ability. This invention uses three edge weight modeling methods and merges the edge weights obtained from the three methods as the final edge weight between graph nodes.

[0071] The state transition edge weights are determined based on the reachability probability of moving from one node state to another under the current policy. The magnitude of the state transition edge weight is positively correlated with the reachability level; that is, a higher edge weight indicates that the policy is more likely to move from one node state to another in the environment. i To another node j This weight can dynamically reflect the exploration path under the current strategy, thus facilitating subsequent path planning guided by the strategy on the graph. By assigning a node to a specific weight under the current strategy... i status As a starting point and another node j status As a sub-goal ( Simulates multi-step execution to compute (approximate) nodes. i and nodes j State transition edge weights between The formula is as follows:

[0072] ;

[0073] In the formula, Indicates that no target is specified. In this case, the lower-level intelligent agent executes t At each time step, it can eventually reach... The number of simulations; Indicates that the specified target is Under these circumstances, it will eventually be possible to reach The number of simulations.

[0074] The energy distance edge weights are determined based on the Euclidean distance between the state spaces of the nodes and the energy required for the action. These edge weights reflect the "energy cost" of moving from one state to another, comprehensively considering both state distance and action cost, making them suitable for physical environments (such as robotic arms, mobile robots, etc.). The formula for expressing the energy transfer edge weights is as follows:

[0075] ;

[0076] In the formula, Indicates node state and node status The Euclidean distance between them; Indicates the state of the slave node to state The energy required for the action; To adjust the hyperparameters for state difference and the proportion of motion energy; Represents the L2 norm; Indicates node state and node status The energy distance between the edges.

[0077] Energy-distance edge weight modeling is suitable for evaluating task execution efficiency and provides a more natural characterization of path unobstructedness in actual motion tasks.

[0078] The neural network predicts edge weights based on a trained scoring model, by introducing state triples. This model enhances the representational capabilities of graphs by automatically learning the semantic edge quality between state points using a data-driven approach. The scoring model takes a triple consisting of a node state pair and an ideal target as input, and performs nonlinear mapping and feature fusion through a multilayer perceptron structure to generate semantic scores representing the reachability, task relevance, and success probability between node state pairs. The scoring results are normalized to the [0,1] interval using a sigmoid function and then used for edge construction and weight assignment in the graph, improving the expressive power and policy interpretability of the graph structure in complex tasks.

[0079] The training samples for the scoring model include positive and negative samples. Positive samples are continuous state pairs extracted from actual trajectory data, corresponding to feasible path points where the strategy has been executed. Negative samples are discontinuous state pairs extracted from failed trajectories, and binary classification labels are constructed from them. , is represented as:

[0080] .

[0081] The following loss function is used when training the scoring model:

[0082] ;

[0083] In the formula, Indicates loss; This indicates that the scoring model is for triples. The edge weight prediction is the normalized result of the semantic score output by the multilayer perceptron structure; where... dg tRepresenting the ideal target. The multilayer perceptron structure includes: an input layer (3D), hidden layer 1 (Linear+ReLU, 128D), hidden layer 2 (Linear+ReLU, 64D), and an output layer (Linear+Sigmoid). The formula for expressing the predicted edge weights of the neural network is:

[0084] ;

[0085] In the formula, Represents a node i and nodes j Neural network prediction of edge weights between them; This represents the scoring process of a multilayer perceptron network structure. The sigmoid function is used for normalization. The neural network predicts edge weights and can automatically learn semantics such as "difficulty," "risk," and "success rate," enhancing the task adaptability of the graph.

[0086] Finally, in the actual construction diagram G =( V , E When ), the formula for calculating the total edge weight between nodes is:

[0087] ;

[0088] In the formula, α , β and γ These are the weights corresponding to the state transition edge weight, energy distance edge weight, and neural network prediction edge weight, respectively. α + β + γ =1.

[0089] S4. When the total edge weight between two nodes is not less than a preset threshold. At that time, that is The corresponding edges are retained and the total edge weight is added to the edges to form a graph structure with semantic edge weights, thereby applying the graph structure to policy training and path planning in hierarchical reinforcement learning.

[0090] It should be noted that not every edge needs to be preserved between nodes; an edge is only preserved when the connection between two nodes is strong enough or has a significant semantic association. This invention combines the advantages of state transition edge weights, energy distance edge weights, and neural network prediction edge weights by weighting and fusing them: state transition edge weights reflect the feasibility and probability of state transitions in the real environment and have clear physical meaning; energy distance edge weights measure the "cost distance" between states and are suitable for sparse reward scenarios; and neural network prediction edge weights can capture complex, nonlinear, or task-related semantics and are adaptable to non-Euclidean spaces.

[0091] To verify the effectiveness of the edge-weight fusion mechanism of this invention, a set of comparative experiments was designed in the AntMazeSmall environment. The experiments compared the training performance of the agent under five graph construction methods. The five graph construction methods are as follows:

[0092] Method 1, State Transition Probability (blue line): calculate edge weights using only state transition probabilities;

[0093] Method 2, Unweighted Modeling (orange line): The graph structure does not model edge weights, which is equivalent to applying a uniform weight to all edges;

[0094] Method 3, Energy Distance (green): Models edge weights using only energy consumption distance;

[0095] Method 4, Neural Prediction (purple line): Only neural networks are used to predict and model edge weights;

[0096] Method 5, Hybrid Weighted Modeling (red line): Fusion of three edge weights (state transition, energy distance, neural prediction) to generate the final total edge weight.

[0097] Please see Figure 2 In the graph, the horizontal axis represents the number of training time steps, and the vertical axis represents the success rate of the robot achieving its goal. The experimental results fully demonstrate that the edge-weighted modeling mechanism of this invention is significantly superior to any single modeling method, and it is remarkably effective in improving the semantic accuracy of graph structures, promoting policy learning efficiency, and increasing task success rate. Especially in multi-stage decision-making tasks (such as AntMaze), this graph modeling strategy based on edge-weighted quality fusion can provide stronger support for sub-goal selection and path planning.

[0098] Example 2

[0099] This embodiment provides a computer terminal, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the graph hierarchical reinforcement learning method based on edge weight modeling as described in Embodiment 1.

[0100] like Figure 3 As shown, the computer terminal provided in this embodiment includes: at least one processor 101, and a memory 102 connected to at least one processor 101. This embodiment does not limit the specific connection medium between the processor 101 and the memory 102. Figure 3The example shown is the connection between processor 101 and memory 102 via bus 100. Bus 100 is... Figure 3 The connections between other components are shown in bold lines and are for illustrative purposes only, not as limiting information. Bus 100 can be divided into address bus, data bus, control bus, etc., for ease of representation. Figure 3 The bus is represented by a single thick line, but this does not indicate that there is only one bus or one type of bus. Alternatively, the processor 101 may also be called a controller; there is no restriction on the name.

[0101] In this embodiment, the memory 102 stores instructions that can be executed by at least one processor 101. The at least one processor 101 can execute the aforementioned method by executing the instructions stored in the memory 102.

[0102] The processor 101 is the control center of the device. It can connect to various parts of the control device through various interfaces and lines. By running or executing instructions stored in memory 102 and calling data stored in memory 102, the processor can perform various functions and process data, thereby monitoring the device as a whole.

[0103] In one possible design, processor 101 may include one or more processing units. Processor 101 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may also not be integrated into processor 101. In some embodiments, processor 101 and memory 102 may be implemented on the same chip; in some embodiments, they may also be implemented on separate chips.

[0104] Processor 101 can be a general-purpose processor, such as a central processing unit (CPU), digital signal processor, application-specific integrated circuit, field-programmable gate array, or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component, capable of implementing or executing the methods, steps, and logic block diagrams disclosed in the embodiments. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the graph hierarchical reinforcement learning method based on edge weight modeling disclosed in Embodiment 1 can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules in processor 101.

[0105] Memory 102, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules. Memory 102 may include at least one type of storage medium, such as flash memory, hard disk, multimedia card, card-type memory, random access memory (RAM), static random access memory (SRAM), programmable read-only memory (PROM), read-only memory (ROM), and electrically erasable programmable read-only memory (EPR). Only Memory (EEPROM), magnetic storage, magnetic disk, optical disk, etc. Memory 102 can be any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but is not limited thereto. In this embodiment, memory 102 can also be a circuit or any other device capable of implementing storage functions for storing program instructions and / or data.

[0106] By designing and programming the processor 101, the code corresponding to the graph hierarchical reinforcement learning method based on edge weight modeling described in the foregoing embodiments can be embedded into the chip, thereby enabling the chip to execute the code during runtime. Figure 1 The steps of the graph hierarchical reinforcement learning method based on edge weight modeling are shown. How to design and program the processor 101 is a technique well-known to those skilled in the art and will not be described further here.

[0107] Example 3

[0108] This embodiment provides a computer-readable storage medium storing a computer program thereon. When the program is executed by a processor, it implements the steps of the graph hierarchical reinforcement learning method based on edge weight modeling as described in Embodiment 1.

[0109] The computer-readable storage medium may include flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the storage medium may be an internal storage unit of a computer device, such as the hard disk or memory of the computer device. In other embodiments, the storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, smart memory card, secure digital card, flash memory card, etc., provided on the computer device. Of course, the storage medium may include both internal storage units and external storage devices of the computer device. In this embodiment, the memory is typically used to store the operating system and various application software installed on the computer device. In addition, the memory can also be used to temporarily store various types of data that have been output or will be output.

[0110] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A graph hierarchical reinforcement learning method based on edge weight modeling, characterized in that, A method for constructing a graph structure applied to policy training and path planning of a mechanical arm or a mobile robot in hierarchical reinforcement learning, the method comprising the following steps: S1. Randomly sample a first candidate node set from the trajectory data in the lower buffer pool and a second candidate node set from the trajectory data in the upper buffer pool; wherein the number of trajectories in the lower buffer pool and the upper buffer pool is the same; each trajectory in the lower buffer pool contains T1 time steps, and the format of each time step is , a number of time steps are sampled from the lower buffer pool as the first candidate node set; each trajectory in the upper buffer pool contains T2 time steps, and the format of each time step is ; wherein T1 > T2, represents a node state, represents a target achieved by a lower-level agent, represents a sub-target, represents an executed action, represents an ideal target; the upper buffer pool is used to preferentially retain key states in a task, and the key states include bifurcated intersections in a maze and grasping moments in robot operation; S2. Selecting a number of target nodes in the first candidate node set and the second candidate node set to form a graph structure node set; S3. Calculating state transition edge weight, energy distance edge weight and neural network prediction edge weight for any two nodes in the graph structure node set, and performing weighted fusion on the three edge weights to calculate total edge weight value; wherein the state transition edge weight is determined based on the reachable probability from one node state to another node state under the current policy; the energy distance edge weight is determined based on the Euclidean distance of the state space between nodes and the energy synthesis of the required action; the neural network prediction edge weight is determined based on a trained scoring model, which takes a triple consisting of a node state pair and an ideal target as input, and performs nonlinear mapping and feature fusion through a multilayer perception network structure to generate a semantic scoring value representing the reachability, task relevance and execution success probability between the node state pair; S4. When the total edge weight value between two nodes is not less than a preset threshold, the corresponding edge is retained and the total edge weight value is added to the edge to form a graph structure with semantic edge weight.

2. The graph hierarchical reinforcement learning method based on edge weight modeling according to claim 1, wherein, In step S2, the target nodes are selected based on a preset weighted farthest point sampling algorithm, and the specific process is as follows: S21. An array is used to save the result set, and the node state weights in the first candidate node set and the second candidate node set are set: wherein is a state of a node i , represents a weight of a state of a node ; represents a first candidate node set represents a second candidate node set > 1; S22. Randomly select one node from the first candidate node set and the second candidate node set i As the initial node, the weight distribution of all candidate node states is calculated at each time of selecting the next node, and the expression formula is: wherein N denotes the set of all nodes except the one that has been selected, j is the set N of any node; denotes the weight distribution over the states N of each node in the set Softmax[·] is the normalized exponential function; denotes the distance between the state of a node and the state of another node; denotes the weight of the state of a node. S23. The node with the maximum weight is selected from the weight distribution and added to the result set until the number of arrays reaches the total number of nodes in the graph structure node set.

3. The graph hierarchical reinforcement learning method based on edge weight modeling according to claim 2, characterized in that, In step S22, the node state The distance between the node state is calculated by the following formula: wherein is a discount in reinforcement learning, denotes a node state after taking an action, reaching a node state is the expected return from the node state denotes the action chosen by the policy during the process from the node state to the node state is output by the policy network.

4. The graph hierarchical reinforcement learning method based on edge weight modeling according to claim 1, wherein, In step S3, a node is set up under the current policy. i status As a starting point and another node j status As a sub-objective, multi-step execution is simulated to compute nodes. i and nodes j State transition edge weights between The weight of the state transition edge is positively correlated with the level of reachability, expressed by the formula: In the formula, represents the number of simulations that can eventually reach when the target is not specified; t represents the number of simulations that can eventually reach when the target is specified. represents the number of simulations that can eventually reach when the target is not specified. represents the number of simulations that can eventually reach 5. The graph hierarchical reinforcement learning method based on edge weight modeling according to claim 4, characterized in that, In step S3, the expression formula of the energy transition edge weight is: wherein, represents the Euclidean distance between node states and node states ; represents the energy of actions required from node states to states ; is a hyperparameter that regulates the difference between states and the proportion of action energy; represents the two-norm; represents the energy distance edge weight between node states and node states ; 6. The graph hierarchical reinforcement learning method based on edge weight modeling according to claim 5, characterized in that, In step S3, the samples for training the scoring model include positive samples and negative samples; wherein the positive samples are continuous state pairs extracted from the actual trajectory data, corresponding to the feasible path points where the strategy has been executed; the negative samples are non-continuous state pairs extracted from the failed trajectories, and a binary classification label is constructed , which is expressed as: 。 7. The graph hierarchical reinforcement learning method based on edge weight modeling according to claim 6, characterized in that, When training the scoring model, the following loss function is used: In the formula, represents the loss; represents the edge weight prediction of the triplet by the scoring model, that is, the result of the semantic scoring output by the multi-layer perception network structure after normalization; wherein dg t represents the ideal target; The expression formula of the neural network prediction edge weight is: In the formula, denotes a node i and a neural network prediction edge weight between nodes j ; denotes a score processing of a multi-layer perception network structure; denotes a normalization processing; The calculation formula of the total edge weight value is: wherein, α , β and γ are the weights corresponding to the state transition edge weight, the energy distance edge weight and the neural network prediction edge weight, respectively.

8. A computer terminal comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, the steps of the graph hierarchical reinforcement learning method based on edge weight modeling according to any one of claims 1-7 are implemented.

9. A computer-readable storage medium having stored thereon a computer program, characterized in that, When the processor executes the program, the steps of the graph hierarchical reinforcement learning method based on edge weight modeling according to any one of claims 1-7 are implemented.

Images