Reinforced learning navigation method and system based on trajectory behavior measurement

Abstract

The invention discloses a reinforcement learning navigation method and system based on track behavior measurement, and relates to the technical field of robot navigation. According to the method, the reward difference, the state transition similarity and the future trajectory representation difference are comprehensively evaluated through trajectory-level behavior measurement, the capturing ability of the robot on a long-term behavior mode is enhanced, and short-looking decision making is avoided; a key state identifier and a mask representation mechanism are introduced, the state importance is dynamically weighted, the key state is accurately positioned, and the number of invalid exploration steps is reduced; by utilizing a forward dynamics model, the modeling capability of state transition dynamics is enhanced, representation collapse is prevented, the strategy stability is improved, the failure rate is reduced, overfitting is avoided, and the generalization capability is enhanced; the exploration direction is guided through a trajectory return estimator in combination with reality and pseudo rewards, strategy convergence is accelerated, and the engineering cost is reduced; parallel environment interaction and batch data sampling are adopted, the encoder is represented in a lightweight manner, the calculation overhead is reduced, and the resource utilization rate is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of robot navigation, and particularly to a reinforcement learning navigation method and system based on trajectory behavior metrics. Background Art

[0002] In the past few years, deep reinforcement learning technology has achieved remarkable results in multiple fields, especially in tasks such as robot navigation, intelligent decision-making, and path planning. The robot navigation task involves the autonomous movement of a robot in an unknown or dynamic environment to reach a target area and avoid obstacles such as collisions. In recent years, navigation algorithms in the field of robotics have gradually introduced reinforcement learning methods to overcome the limitations in traditional navigation strategies, especially the adaptability and flexibility issues in complex environments. The exploration problem in robot navigation is particularly prominent, especially when the navigation task depends on the robot continuously exploring new environmental areas. Traditional reinforcement learning methods based on external rewards perform poorly in environments with sparse rewards or without immediate feedback, making it a difficult problem to effectively guide the exploration of robots in the face of environmental uncertainties.

[0003] In recent years, the intrinsic motivation method has gradually become a dominant exploration strategy, which motivates reinforcement learning agents to make decisions based on their own satisfaction, especially when exploring new states and uncertain regions. Different from traditional methods that rely on direct environmental feedback, the prediction-based intrinsic motivation method uses the predicted state representation to quantify the novelty of the state, thereby guiding the exploration of the agent. In this context, the state representation becomes a key factor affecting the exploration efficiency of the navigation agent and the learning of the optimal strategy. Although existing behavior metric methods provide potential support for the state representation in intrinsic motivation, they still face several significant challenges when applied in dynamic environments: (1) The dynamic nature of the environment and the existence of uninformative rewards may lead to representation collapse, thereby triggering ineffective exploration. (2) In existing methods, behavior metrics are usually evaluated based on single-step metrics, but this method lacks consideration of the long-term dependence and sequence information of the entire navigation trajectory, which may lead to errors in behavior estimation. Summary of the Invention

[0004] The purpose of the present invention is to provide a reinforcement learning navigation method and system based on trajectory behavior metrics to improve the above technical problems.

[0005] To achieve the above invention purpose, the embodiments of the present invention provide the following technical solutions:

[0006] A reinforcement learning navigation method based on trajectory behavior metrics includes:

[0007] S1, obtaining a set of historical state trajectories of the robot;

[0008] S2. Randomly select two historical state trajectories and input them into the reinforcement learning navigation model for optimization to obtain the optimized reinforcement learning navigation model; the reinforcement learning navigation model uses the reinforcement learning algorithm;

[0009] S3. Based on the navigation goal, use the optimized reinforcement learning navigation model to generate the future trajectory of the robot and execute this future trajectory;

[0010] S4. Repeat S3 until the navigation goal is completed.

[0011] Further, the S2 includes:

[0012] S2-1. Take the robot as the agent, initialize the replay buffer and the number of episodes, and randomly generate the initial environment configuration and the agent position;

[0013] S2-2. Randomly select two historical state trajectories;

[0014] S2-3. Obtain the current policy; collect tuple data based on the current policy and store it in the replay buffer; the tuple data includes all states, actions, and rewards; the tuple data includes the states, rewards, and actions corresponding to the two historical state trajectories;

[0015] S2-4. Calculate the trajectory behavior similarity of the two historical state trajectories;

[0016] S2-5. Randomly sample a batch of tuple data from the replay buffer to obtain the tuple sampling data;

[0017] S2-6. Calculate the importance scores, mask representations, and inverted mask representations corresponding to each state in the tuple sampling data;

[0018] S2-7. Based on the trajectory behavior similarity, use the forward dynamics model and the trajectory return estimator to optimize the reinforcement learning navigation model to obtain the initial optimized reinforcement learning navigation model; increment the current episode number by 1;

[0019] S2-8. Repeat S2-6 to S2-7 until the current episode number reaches the number of episodes to obtain the optimized reinforcement learning navigation model; the initial value of the current episode number is 0.

[0020] Further, the S2-3 includes:

[0021] S2-3-1. Obtain the current policy and determine the current state and current action at the current time step;

[0022] S2-3-2. Based on the current policy, execute the current action to obtain the next state and the corresponding reward at the next time step;

[0023] S2-3-3. Store the current state, current action, next state, and reward in the replay buffer; increment the time step by 1; the initial value of the time step is 0;

[0024] S2-3-4. Repeat S2-3-1 to S2-3-4 until the time step reaches the maximum time step. Treat all states, actions, and rewards as tuple data and store them in the replay buffer.

[0025] Further, the S2-4 includes:

[0026] S2-4-1. Calculate the importance corresponding to each state in the tuple sampling data through the state identification function and perform weighting to obtain the corresponding importance scores;

[0027] S2-4-2. Based on the importance scores, calculate the masked representation and the inverse masked representation of the tuple sampling data through the encoder;

[0028] S2-4-3. Calculate the trajectory behavior similarity of two historical state trajectories based on the masked representation and the inverse masked representation.

[0029] Further, the S2-7 includes:

[0030] S2-7-1. Based on the reward and the masked representation and the inverse masked representation in S2-6, use the trajectory return estimator to calculate the corresponding masked loss function and inverse masked loss function;

[0031] S2-7-2. Calculate the overall objective loss function based on the masked loss function and the inverse masked loss function;

[0032] S2-7-3. Based on the overall objective loss function, adjust the parameters of the reinforcement learning navigation model to obtain the adjusted reinforcement learning navigation model;

[0033] S2-7-4. Train the forward dynamic model to obtain the trained forward dynamic model;

[0034] S2-7-5. Use the trained forward dynamic model to generate the next states corresponding to two historical state trajectories and generate the corresponding future masked representation;

[0035] S2-7-6. Calculate the trajectory-level behavior metric loss function based on the rewards, future masked representation, and trajectory behavior similarity corresponding to two historical state trajectories;

[0036] S2-7-7. Update the parameters of the adjusted reinforcement learning navigation model based on the trajectory-level behavior metric loss function to obtain the initial optimized model of the reinforcement learning navigation.

[0037] The formula corresponding to S2-7-6 is:

[0038] The formula corresponding to S2-7-1 is as follows:

[0039]

[0040] where l masked and l inverted represent the mask loss function and the inverted mask loss function respectively, ρ k and ρ k represent the k-th mask representation and the k-th inverted mask representation corresponding to the tuple sampling data respectively, R k represents the k-th reward, and f TRE (·) represents the trajectory return estimator.

[0041] Further, the formula corresponding to S2-7-6 is as follows:

[0042]

[0043] where tr i and tr j represent two selected historical state trajectories respectively, R i and R j represent respectively, l metric represents the trajectory-level behavior metric loss function, |·| represents the absolute value, ρ′ i and ρ′ j represent the future mask representations corresponding to the historical state trajectory tr i and the historical state trajectory tr j respectively, d π (tr i , tr j ) represents the trajectory behavior similarity between the historical state trajectory tr i and the historical state trajectory tr j , and E(·) represents the expected value of a random variable. The behavior metric is based on the average behavior of a set of trajectories.

[0044] A reinforcement learning navigation system based on trajectory behavior metric includes:

[0045] A robot historical state trajectory acquisition module, configured to obtain a set of robot historical state trajectories;

[0046] A reinforcement learning navigation model optimization module, configured to randomly select two historical state trajectories and input them into the reinforcement learning navigation model for optimization to obtain a reinforced learning navigation optimization model; the reinforcement learning navigation model adopts a reinforcement learning algorithm;

[0047] A reinforcement learning navigation module, configured to use the reinforcement learning navigation optimization model to generate a future trajectory of the robot and execute the future trajectory.

[0048] The beneficial effects of the present invention are as follows:

[0049] Through trajectory-level behavior metrics, the present invention comprehensively evaluates reward differences, state transition similarities, and future trajectory representation differences, enhances the robot's ability to capture long-term behavior patterns, avoids short-sighted decision-making, and significantly improves the navigation success rate in complex dynamic scenarios; introduces a key state identifier and a mask representation mechanism, dynamically weights the importance of states, accurately locates key states, reduces the number of ineffective exploration steps, and improves exploration efficiency; utilizes a forward dynamics model to enhance the modeling ability of state transition dynamics, prevent representation collapse, improve policy stability, reduce the failure rate, avoid overfitting, and enhance generalization ability; combines actual and pseudo rewards through a trajectory return estimator to guide the exploration direction, accelerate policy convergence, shorten the training time, reduce the dependence on manually designed intensive rewards, and reduce engineering costs; adopts parallel environment interaction and batch data sampling to improve training throughput, lightweight representation encoder, reduce computational overhead, improve resource utilization, and is applicable to tasks with high real-time requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following will briefly introduce the drawings required in the embodiments. It should be understood that the following drawings only show some embodiments of the present invention, and therefore should not be regarded as limiting the scope. For those of ordinary skill in the art, without creative efforts, other related drawings can also be obtained based on these drawings.

[0051] Figure 1 is the flowchart of the method in the embodiment of the present invention;

[0052] Figure 2 is the schematic diagram of the generation of the mask loss function and the inverted mask loss function in the embodiment of the present invention;

[0053] Figure 3 is the schematic diagram of the generation of the future mask representation in the embodiment of the present invention;

[0054] Figure 4 is the schematic diagram of the ObstructedMaze as the blocked maze task and the KeyCorridor as the key corridor task in the embodiment of the present invention;

[0055] Figure 5(a) is the comparison curve graph of multiple methods under the ObstructedMaze as the blocked maze task in the embodiment of the present invention;

[0056] Figure 5(b) is the comparison curve graph of multiple methods under the KeyCorridor as the key corridor task in the embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0057] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. The components of the embodiments of the present invention described and illustrated herein can be arranged and designed in a variety of different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative efforts belong to the scope of protection of the present invention.

[0058] Please refer to Figure 1 , a reinforcement learning navigation method based on trajectory behavior metrics provided in this embodiment includes:

[0059] S1. Obtain the robot historical state trajectory set TR = [tr1, tr2,..., tr T-1 , tr T ; The time horizon T is a preset time period. Among them, tr1, tr2, tr T-1 , tr T are the 1st historical state trajectory, the 2nd historical state trajectory, the (T - 1)th historical state trajectory, and the Tth historical state trajectory respectively.

[0060] S2. Randomly select two historical state trajectories tr i , tr j and input them into the reinforcement learning navigation model for optimization to obtain the reinforcement learning navigation optimization model; The reinforcement learning navigation model uses the reinforcement learning algorithm;

[0061] Among them, are the 1st historical state, the 2nd historical state, and the Tth historical state of the ith historical state trajectory tr i respectively; are the 1st historical state, the 2nd historical state, and the Tth historical state of the jth historical state trajectory tr j respectively.

[0062] The S2 includes:

[0063] S2-1. Take the robot as an agent, initialize the replay buffer and the number of episodes, and randomly generate the initial environment configuration and the agent position;

[0064] S2-2. Randomly select two historical state trajectories;

[0065] S2-3. Obtain the current policy; collect tuple data based on the current policy and store it in the replay buffer; the tuple data includes all states, actions, and rewards; the tuple data includes the states, rewards, and actions corresponding to two historical state trajectories.

[0066] S2-3 includes:

[0067] S2-3-1. Obtain the current policy, and determine the current state and current action at the current time step.

[0068] S2-3-2. Execute the current action based on the current policy, and obtain the next state and the corresponding reward at the next time step.

[0069] S2-3-3. Store the current state, current action, next state, and reward in the replay buffer; increment the time step by 1; the initial value of the time step is 0.

[0070] S2-3-4. Repeat S2-3-1 to S2-3-4 until the time step reaches the maximum time step, and store all states, actions, and rewards as tuple data in the replay buffer.

[0071] S2-4. Calculate the trajectory behavior similarity of two historical state trajectories.

[0072] S2-4 includes:

[0073] S2-4-1. Calculate the importance of each state in the tuple sampling data through the key state identification function and perform weighting to obtain the corresponding importance score.

[0074] The formula corresponding to S2-4-1 is:

[0075] score k =σ(f PSI (φ(tr k ))), k = i, j;

[0076]

[0077] where score k represents the importance score corresponding to the historical state trajectory tr k , φ(·) represents the state encoder, f PSI (·) represents the key state identification function, and σ(·) represents the sigmoid activation function. are respectively the first state importance score, the second state importance score, and the T-th state importance score corresponding to the historical state trajectory tr k , respectively represent the first state, the second state, and the T-th state corresponding to the historical state trajectory trk.

[0078] The key state identification function f PSI (·) assigns a score to each hidden state representation, effectively reflecting the degree of importance in the context of the trajectory.

[0079] S2-4-2. Based on the importance scores, calculate the masked representation ρk and the inverted masked representation ρ of the tuple sampling data through the encoder k ;

[0080] The formula corresponding to S2-4-2 is:

[0081]

[0082] where ψ(·) represents the sequence encoder, which is used to extract the temporal relationship between tuple sampling data.

[0083] S2-4-3. Based on the masked representation and the inverted masked representation, calculate the trajectory behavior similarity of two historical state trajectories. Trajectories generated by the same policy in the same environment are similar in terms of state changes, action selections, and reward returns. Therefore, calculating the trajectory behavior similarity of two historical state trajectories helps the reinforcement learning navigation model to more accurately understand the distinctions between various states.

[0084] The formula corresponding to the trajectory behavior similarity is:

[0085] d π (tr i ,tr j ) = ||ρ i - ρ j ||2;

[0086] where ||·||2 represents the norm 2, ρ i , ρ j respectively represent the representations of the historical state trajectory tr i and the historical state trajectory tr j , and d π (tr i ,tr j ) represents the trajectory behavior similarity between the historical state trajectory tr i and the historical state trajectory tr j . d π (·) represents a pseudo-metric, which is used to measure the equivalence of the policy π.

[0087] To study the importance of states within a trajectory, a mechanism for distinguishing critical states is introduced, namely the content of S2-4-1 and S2-4-2. Considering critical and unimportant states, a critical state identification function f PSI (·) is introduced, enabling the model to more comprehensively and finely evaluate the contributions of each state, thus facilitating a deeper analysis of the agent's behavior patterns, helping to quantify behavioral similarities, and promoting more profound progress in understanding the environmental structural dynamics.

[0088] S2-5: Sample a batch of tuple data from the random sampling replay buffer to obtain tuple sampling data;

[0089] S2-6: Calculate the importance scores, mask representations, and inverted mask representations corresponding to each state in the tuple sampling data;

[0090] S2-7: Based on the trajectory behavior similarity, use the forward dynamics model and the trajectory reward estimator to optimize the reinforcement learning navigation model to obtain the initial optimized reinforcement learning navigation model; increment the current round number by 1; the trajectory reward estimator is an MLP network. The forward dynamics model can use a linear layer.

[0091] The accuracy of predicting trajectory rewards can be used as one of the criteria for evaluating the effectiveness of trajectory representation learning. Effective trajectory representation learning can successfully capture the subtle dynamics of state estimation. The accuracy of predicting trajectory rewards enables the reinforcement learning agent to make informed decisions, enhance exploration capabilities, and ultimately lead to more efficient and effective learning. On the contrary, ineffective trajectory representation learning fails to cover the complexity of state transitions in the environment. Consequently, it cannot provide accurate reward estimates, which hinders the agent's understanding of potential dynamics and often leads to suboptimal decisions and exploration. Therefore, a trajectory reward estimator is introduced to assist in the learning of the trajectory representation learned by the previous critical state identifier. The trajectory reward estimator and the critical state identification function optimize the model from two different perspectives, accurately estimating the trajectory rewards related to the masked trajectory representation and hindering the inverted mask trajectory representation to prevent it from providing accurate trajectory reward predictions.

[0092] The S2-7 includes:

[0093] S2-7-1: Based on the reward and the mask representation and inverted mask representation in S2-6, use the trajectory reward estimator to calculate the corresponding masked loss function and inverted mask loss function;

[0094] As Figure 2 shown, the formula corresponding to the S2-7-1 is;

[0095]

[0096] where, l masked 、linverted respectively represent the masked loss function and the inverse masked loss function, ρ k and ρ k respectively represent the k-th masked representation and the k-th inverse masked representation corresponding to the tuple sampled data, R k represents the k-th reward, f TRE (·) represents the trajectory return estimator, and ∑(·) represents the summation function; represents the t-th corresponding reward corresponding to the historical state trajectory tr k

[0097] S2-7-2. Calculate the overall objective loss function L based on the masked loss function and the inverse masked loss function TRE ;

[0098] The corresponding formula for S2-7-2 is:

[0099] L TRE = l masked + cl inverted ;

[0100] where η represents the weighted balance between the masked representation and the inverse masked representation.

[0101] This method simultaneously utilizes masked and inverse masked trajectory representations, allowing the trajectory return estimator to obtain information not only from accurate relevant state predictions but also from the intentional interference introduced by the inverse representation, which helps to more comprehensively understand the importance and contribution of each state in the trajectory. Therefore, the finally obtained trajectory representation retains the significant information of individual states while revealing the temporal dependence between states. This enhanced representation supports more detailed modeling and interpretation of long-term behavior patterns. This method helps the agent gain a structural insight into the environment through weighted trajectory representations, going beyond the traditional method of only analyzing single-step state transitions and ultimately achieving a more comprehensive and extensive metric.

[0102] S2-7-3. Adjust the parameters of the reinforcement learning navigation model based on the overall objective loss function to obtain an adjusted reinforcement learning navigation model;

[0103] S2-7-4. Train the forward dynamic model to obtain a trained forward dynamic model;

[0104] According to the formula:

[0105]

[0106] Train the forward dynamic model with the goal of minimizing, and adjust the weight parameters of the forward dynamic model. Among them, l for represents the mean squared error loss, s k ​respectively represent the next predicted state and the state in the sampled tuple data. Among them, when k is i, the object processed by this formula is the historical state trajectory tr i corresponding state; when k is j, the object processed by this formula is the historical state trajectory tr j corresponding state.

[0107] S2-7-5. Generate the next states corresponding to two historical state trajectories using the trained forward dynamic model, and generate the corresponding future mask representations

[0108] As Figure 3 shown, according to the formula:

[0109]

[0110] Generate the next states corresponding to two historical state trajectories using the trained forward dynamic model where a k represents the action, and f FOR (·) represents the forward dynamic model. When k is i, the object processed by this formula is the historical state trajectory tr i corresponding state, and obtain the next state corresponding to the historical state trajectory tr i When k is j, the object processed by this formula is the historical state trajectory tr corresponding state, and obtain the next state corresponding to the historical state trajectory tr j corresponding state, and obtain the next state corresponding to the historical state trajectory tr j corresponding state.

[0111] According to the formula:

[0112]

[0113] Obtain the future mask representations ρ′ i , ρ′ j ; among them, ρ′ k , score k ' respectively represent the future mask representation and the importance score of the next trajectory tr k corresponding to the historical state trajectory tr k ', and represents element-wise multiplication.

[0114] S2-7-6. Calculate the trajectory-level behavior metric loss function based on the rewards, future mask representations, and trajectory behavior similarities corresponding to two historical state trajectories;

[0115] The formula corresponding to S2-7-6 is:

[0116]

[0117] Among them, tr i , tr j respectively represent two selected historical state trajectories, R i , R j respectively represent, l metric represents the trajectory-level behavior metric loss function, |·| represents the absolute value, ρ′ i , ρ′ j respectively represent the future mask representations corresponding to the historical state trajectory tri and the historical state trajectory trj, dπ(tri, trj) represents the trajectory behavior similarity between the historical state trajectory tri and the historical state trajectory tr j . The trajectory behavior similarity, E(·) represents the expected value of a random variable, and the behavior metric is based on the average behavior of a set of trajectories.

[0118] By optimizing the trajectory-level behavior metric loss function, the reinforcement learning navigation model is made to perceive the differences between states.

[0119] S2-7-7. Based on the trajectory-level behavior metric loss function, update the parameters of the adjusted reinforcement learning navigation model to obtain the initial optimized model of the reinforcement learning navigation.

[0120] S2-8. Repeat S2-6 to S2-7 until the current round number reaches the number of rounds to obtain the optimized model of the reinforcement learning navigation; the initial value of the current round number is 0.

[0121] S3. Based on the navigation goal, use the optimized model of the reinforcement learning navigation to generate the future trajectory of the robot and execute the future trajectory.

[0122] S4. Repeat S3 until the navigation goal is completed.

[0123] In summary, through trajectory-level behavior metrics, this method comprehensively evaluates reward differences, state transition similarities, and future trajectory representation differences, enhances the robot's ability to capture long-term behavior patterns, avoids short-sighted decisions, and significantly improves the navigation success rate in complex dynamic scenarios; introduces a key state identifier and a mask representation mechanism, dynamically weights the importance of states, accurately locates key states, reduces the number of ineffective exploration steps, and improves exploration efficiency; uses a forward dynamics model to enhance the modeling ability of state transition dynamics, prevent representation collapse, improve policy stability, reduce the failure rate, avoid overfitting, and enhance generalization ability; combines actual and pseudo-rewards through a trajectory return estimator to guide the exploration direction, accelerate policy convergence, shorten the training time, reduce the dependence on manually designed dense rewards, and reduce engineering costs; adopts parallel environment interaction and batch data sampling to improve training throughput, lightweight the representation encoder, reduce computational overhead, improve resource utilization, and is applicable to tasks with high real-time requirements.

[0124] A reinforcement learning navigation system based on trajectory behavior metrics includes:

[0125] A robot historical state trajectory acquisition module, which is used to obtain a set of robot historical state trajectories;

[0126] A reinforcement learning navigation model optimization module, which is used to randomly select two historical state trajectories and input them into the reinforcement learning navigation model for optimization to obtain a reinforced learning navigation optimization model; the reinforcement learning navigation model adopts a reinforcement learning algorithm;

[0127] A reinforcement learning navigation module, which is used to generate a future trajectory of the robot by using the reinforcement learning navigation optimization model and execute the future trajectory.

[0128] This system uses the reinforcement learning navigation model optimization model for robot navigation, enhancing the robot's ability to capture long-term behavior patterns; it has a simple structure and requires few computing resources, reducing engineering costs. By adopting parallel environment interaction and batch data sampling, it improves the training throughput and reduces the computing overhead.

[0129] In this embodiment, the sampling simulation environment MiniGrid is used for task testing. As Figure 4 shown, ObstructedMaze is selected as the blocked maze task and KeyCorridor is selected as the key corridor task. MiniGrid is a set of procedurally generated grid world environments with sparse rewards. At the beginning of each episode, the configuration of the grid and the initial position of the agent (robot) are randomly initialized. These grids are characterized by grids that contain a single object or no object, and may contain various types of objects, including walls, doors, keys, balls, boxes, and goals. For example, when a door may be blocked by a ball, the agent needs to move the ball first to unlock the door. Each object is distinguished by a different color. The goal of the agent is to navigate the entire grid, avoid colliding with obstacles, collect useful items, and finally reach the specified target position. When the task is successfully completed, the agent usually obtains a positive reward, while when it fails, it will obtain a reward of 0.

[0130] The formula corresponding to the sparse reward is:

[0131] r = 1 - 0.9t / t max ;

[0132] where r represents the sparse reward, t represents the time steps taken, and t max represents the maximum number of time steps.

[0133] Observations in the environment are a 7×7×3 partial observation, representing attributes including tuple object type, color, and door status (e.g., 0 means the door is open, 1 means the door is closed, 2 means the door is locked). There are seven discrete actions that make up the action space: turn left, turn right, move forward, pick up an object, drop the carried object, open the door, and interact with the object. The agent can only carry one item at a time (e.g., a ball or a key). To open a locked door, the agent needs to carry a key that matches the color of the door.

[0134] The hyperparameter settings in the experiment are shown in Table 1.

[0135] Table 1

[0136] Hyperparameter Value Number of parallel environments 128 Number of interactions per episode 128 Value function loss coefficient 0.5 Learning rate 0.0005 Discount factor 0.99 Gradient clipping coefficient 0.5 Forward loss coefficient 10 Backward loss coefficient 0.1

[0137] The present invention is compared with intrinsic reward methods based on the number of accesses and counts (PPO, ICM, RIDE, RND, Novel). The comparison results are shown in Figures 5(a) and 5(b). The present invention is superior to other intrinsic reward methods in terms of effectiveness and efficiency.

[0138] The above are only the specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present invention can easily think of changes or substitutions, which should be covered by the protection scope of the present invention. Therefore, the protection scope of the present invention should be determined by the protection scope of the claims.

Claims

1. A reinforcement learning navigation method based on trajectory behavior metrics, characterized in that, Including: S1. Obtain the set of historical state trajectories of the robot; S2. Randomly select two historical state trajectories and input them into the reinforcement learning navigation model for optimization to obtain the optimized reinforcement learning navigation model; the reinforcement learning navigation model uses the reinforcement learning algorithm; S3. Based on the navigation goal, use the optimized reinforcement learning navigation model to generate the future trajectory of the robot and execute the future trajectory; S4. Repeat S3 until the navigation goal is completed.

2. The reinforcement learning navigation method based on trajectory behavior metric according to claim 1, wherein The S2 includes: S2-1. Take the robot as the agent, initialize the replay buffer and the number of episodes, and randomly generate the initial environment configuration and the agent position; S2-2. Randomly select two historical state trajectories; S2-3. Obtain the current policy; collect tuple data based on the current policy and store it in the replay buffer; the tuple data includes all states, actions, and rewards; the tuple data includes the states, rewards, and actions corresponding to the two historical state trajectories; S2-4. Calculate the trajectory behavior similarity of the two historical state trajectories; S2-5. Randomly sample a batch of tuple data from the replay buffer to obtain the tuple sampling data; S2-6. Calculate the importance scores, mask representations, and inverted mask representations corresponding to each state in the tuple sampling data; S2-7. Based on the trajectory behavior similarity, use the forward dynamics model and the trajectory return estimator to optimize the reinforcement learning navigation model to obtain the initial optimized reinforcement learning navigation model; increment the current episode number by 1; S2-8. Repeat S2-6 to S2-7 until the current episode number reaches the number of episodes to obtain the optimized reinforcement learning navigation model; the initial value of the current episode number is 0.

3. The reinforcement learning navigation method based on trajectory behavior metric according to claim 2, characterized in that The S2-3 includes: S2-3-1. Obtain the current policy, determine the current state and the current action at the current time step; S2-3-2. Execute the current action based on the current policy to obtain the next state and the corresponding reward at the next time step; S2-3-3. Store the current state, current action, next state, and reward in the replay buffer; increment the time step by 1; the initial value of the time step is 0; S2-3-4. Repeat S2-3-1 to S2-3-4 until the time step reaches the maximum time step, and use all states, actions, and rewards as tuple data and store them in the replay buffer.

4. The reinforcement learning navigation method based on trajectory behavior metric according to claim 2, wherein The S2-4 includes: S2-4-1. Calculate the importance corresponding to each state in the tuple sampling data through the key state identification function and perform weighting to obtain the corresponding importance scores; S2-4-2. Based on the importance scores, calculate the mask representation and the inverted mask representation of the tuple sampling data through the encoder; S2-4-3. Calculate the trajectory behavior similarity of the two historical state trajectories based on the mask representation and the inverted mask representation.

5. The reinforcement learning navigation method based on trajectory behavior metric according to claim 2, characterized in that The S2-7 includes: S2-7-1. Based on the reward and the mask representation and the inverted mask representation in S2-6, use the trajectory return estimator to calculate the corresponding mask loss function and inverted mask loss function; S2-7-2. Calculate the overall objective loss function based on the mask loss function and the inverted mask loss function; S2-7-3. Adjust the parameters of the reinforcement learning navigation model based on the overall objective loss function to obtain an adjusted reinforcement learning navigation model; S2-7-4. Train the forward dynamic model to obtain a trained forward dynamic model; S2-7-5. Use the trained forward dynamic model to generate the next states corresponding to two historical state trajectories and generate corresponding future mask representations; S2-7-6. Calculate the trajectory-level behavior metric loss function based on the rewards, future mask representations, and trajectory behavior similarities corresponding to two historical state trajectories; S2-7-7. Update the parameters of the adjusted reinforcement learning navigation model based on the trajectory-level behavior metric loss function to obtain an initial optimized reinforcement learning navigation model.

6. The reinforcement learning navigation method based on trajectory behavior metric according to claim 5, characterized in that The formula corresponding to S2-7-1 is: where, l masked and l inverted represent the masked loss function and the inverse masked loss function respectively, ρ k and ρ k represent the k-th masked representation and the k-th inverse masked representation corresponding to the tuple sampling data respectively, R k represents the k-th reward, and f TRE (·) represents the trajectory return estimator.

7. The reinforcement learning navigation method based on trajectory behavior metric according to claim 5, wherein The formula corresponding to S2-7-6 is: Among them, tr i and tr j respectively represent two selected historical state trajectories, R i and R j respectively represent, l metric represents the trajectory-level behavior metric loss function, |·| represents the absolute value, ρ' i and ρ' j respectively represent the future mask representations corresponding to the historical state trajectory tr i and the historical state trajectory tr j , d π (tr i , tr j ) represents the trajectory behavior similarity between the historical state trajectory tr i and the historical state trajectory tr j . E(·) represents the expected value of a random variable, and the behavior metric is based on the average behavior of a set of trajectories.

8. A reinforcement learning navigation system based on trajectory behavior metrics is used to implement a reinforcement learning navigation method based on trajectory behavior metrics according to any one of claims 1 to 7, characterized in that, It includes: A robot historical state trajectory acquisition module for obtaining a set of robot historical state trajectories; A reinforcement learning navigation model optimization module for randomly selecting two historical state trajectories and inputting them into the reinforcement learning navigation model for optimization to obtain an optimized reinforcement learning navigation model; the reinforcement learning navigation model uses a reinforcement learning algorithm; A reinforcement learning navigation module for using the optimized reinforcement learning navigation model to generate a future trajectory of the robot and execute the future trajectory.

Images