An indoor robot vision navigation method based on optimal policy replay

Abstract

The application discloses an indoor robot vision navigation method based on an optimal strategy playback, which is divided into an on-policy algorithm and an off-policy algorithm, current navigation targets are trained in the on-policy, and old navigation targets are reviewed in the off-policy to reduce catastrophic forgetting. The method comprises a single feature extraction network and a strategy network. The input of the strategy network is the output of the state feature extraction network, namely, the advanced feature map of a current image. The output of the whole network is two functions: a strategy function pi (a|s) and a value function Q (s|a). The whole training process is divided into two stages: an on-policy stage and an off-policy stage. In the on-policy stage, an agent learns the strategy of a new target through interaction with an environment, and in the off-policy stage, the agent reviews the learned strategy by using the optimal experience of old targets stored in a memory to prevent catastrophic forgetting. The application utilizes the experience in the memory to learn the experience of old navigation targets, so that forgetting can be reduced to the maximum.

Description

Technical Field

[0001] This invention belongs to the field of navigation technology, specifically relating to an indoor robot visual navigation method. Background Technology

[0002] Visual navigation is a fundamental problem in robotics and artificial intelligence. Goal-driven visual navigation tasks involve commanding a robot to search for a given object within a visual scene. The object description can be visual or semantic information, and the robot's input is the current visual image. In recent years, visual navigation has attracted increasing research interest from the fields of artificial intelligence and computer vision. Indoor robot visual navigation has been applied in numerous fields, including automated home services, warehouse and logistics management, and the hotel and tourism industries.

[0003] Traditional approaches are map-based visual navigation methods. These methods explicitly decompose the navigation task into a set of sub-tasks: mapping, localization, planning, and motion control. While these methods have achieved considerable success over the years, their modular design has fundamental limitations that hinder their widespread adoption. A significant limitation is their susceptibility to sensor noise accumulating from the mapper and propagating to the controller, leading to error accumulation and making these algorithms less robust in complex environments. More importantly, they require extensive case-specific and scenario-driven hand-engineering, making them difficult to integrate with other downstream AI tasks. Due to the success of learning-based methods, particularly reinforcement learning-based methods, in relevant tasks in recent years, there has been a surge in work applying them to indoor visual navigation. These learning-based methods typically take visual input and a task-specified navigation goal as input, and output the optimal action taken by the agent at each timetamp to achieve the task-specified goal. Unlike traditional methods, learning-based methods infer solutions directly from the current input, making them an end-to-end approach. Therefore, they require minimal hand-engineering and can serve as the foundation for new AI-driven visual navigation tasks. However, current learning-based visual navigation methods typically employ a training approach that randomly selects a navigation target at the start of each training task when learning multiple navigation targets. While this allows the network model to learn multiple targets simultaneously, it lacks flexibility because whenever a new target needs to be learned, it must be combined with previously learned targets and relearned. This process not only wastes significant resources but also limits the applicability of learning-based navigation algorithms. This places high demands on the model's stability and adaptability. Existing methods such as EWC and UNCLEAR have not performed ideally in addressing this issue. Summary of the Invention

[0004] To overcome the shortcomings of existing technologies, this invention provides an indoor robot visual navigation method based on optimal policy playback, consisting of an on-policy algorithm and an off-policy algorithm. Catastrophic forgetting is reduced by training the current navigation target in the on-policy phase and reviewing old navigation targets in the off-policy phase. This method employs a single feature extraction network and a policy network. The input to the policy network is the output of the state feature extraction network, i.e., the high-level feature map of the current image. The output of the entire network is two functions: a policy function π(a|s) and a value function Q(s|a). The entire training process is divided into two phases: an on-policy phase and an off-policy phase. In the on-policy phase, the agent learns the policy for new targets through interaction with the environment, while in the off-policy phase, the agent uses the optimal experience of old targets stored in memory to review the learned policy, preventing catastrophic forgetting. This invention effectively utilizes the experience stored in memory to learn the experience of old navigation targets, minimizing forgetting.

[0005] The technical solution adopted by this invention to solve its technical problem includes the following steps:

[0006] Step 1: Virtual Environment Preparation

[0007] In a virtual environment, a navigation task includes a scene S, an initial point p, and a navigation target o; the goal of the agent is to find the navigation target o in the 3D environment from the initial position within a given number of steps; the agent's action space is limited to six actions: move forward, turn left, turn right, look down, look up, and complete.

[0008] In each step, the robot receives an image from the current camera in the scene and acts in one of two ways: 1) selects an action and moves accordingly, or 2) selects to complete the action and terminate the task; the set of all steps from the start to the end of the task is called a trajectory episode; the task is considered successful if the following four conditions are met simultaneously: 1) the robot selects to complete the action; 2) the robot is no more than 1 meter away from the given target object; 3) the target object is within the robot's field of vision; 4) the robot does not exceed the maximum number of steps.

[0009] Multiple navigation targets were set up. During training, 1 million trajectories were trained for each navigation target, and the training order was the same as that of the navigation targets.

[0010] Step 2: Robot visual navigation;

[0011] Step 2-1: On-policy learning;

[0012] In the on-policy phase, the goal is to train the agent to learn the optimal policy for the current target. When learning a new navigation policy, the agent interacts directly with the environment, and the policy gradient is given by the following formula:

[0013]

[0014] Where θ is the parameter of the neural network, and π θ (a t |s t This is the current strategy. It is an estimation of the action value function. It is an estimate of the value function, where k is the number of actions, T is the total number of steps in the trajectory, and a t The action chosen by the agent at time t, s t It is the state of the agent at time t;

[0015] Step 2-2: Off-policy learning

[0016] In the off-policy phase, experience is collected when the number of training trajectories for a navigation target reaches its maximum value; the trajectories stored in memory are represented as episode = {(s t ,a t ,r t ,π t Q t )|1≤t≤T}, target=(target1,target2,…,target n ), where T is the total number of steps in the trajectory, and m is the size of the memory; the expression for memory is as follows:

[0017]

[0018] Where, r t The reward obtained by the agent at time t, π t The agent's policy at time t, Q t It is the agent's Q-function at time t;

[0019] From the perspective of memory storage strategies, the strategies for storing data in memory are:

[0020] Memory = (π1,π2,π3,…,π) n (3)

[0021] Where (π1,π2,π3,…,π) n ) represents (target1, target2, ..., target n The optimal strategy;

[0022] Step 3: The off-policy phase is performed multiple times after the policy learning phase; RetraceQ is used. ret (x t ,a t To estimate Q π (a t |s t ) and bias-corrected truncation importance sampling is used to reduce the variance of off-policy distribution drift; where Q ret (x t ,a t The formula is as follows:

[0023]

[0024] The optimization objectives for off-policy are as follows:

[0025]

[0026] in, It is the importance weight of the truncation. in [x] + =x, c is a constant, and E(a~π) is the expectation of strategy π;

[0027] Based on the above optimization objectives, KL and L2 losses are further used to constrain the difference between the target policy and the behavioral policy.

[0028] Preferably, the virtual environment is an AI2-THOR virtual environment.

[0029] Preferably, the robot receives an image from the current camera in the scene, and the image size is 300*300*3.

[0030] Preferably, the navigation targets are set to 10, namely: "alarm clock", "book", "bowl", "coffee machine", "kettle", "plate", "frying pan", "toaster", "pot", and "refrigerator".

[0031] The beneficial effects of this invention are as follows:

[0032] 1. This invention divides the training phase of the navigation algorithm into on-policy learning and off-policy learning. On-policy learning only learns the experience of the current navigation target, while off-policy learning only learns the experience of previous navigation targets. This maximizes the utilization of memory-based experience to learn the experience of previous navigation targets.

[0033] 2. This invention stores only the optimal strategy trajectory in memory, which can minimize forgetting.

[0034] 3. This invention can maintain good anti-forgetting properties even in very small amounts of memory. Attached Figure Description

[0035] Figure 1 This is a schematic diagram of the network model of the present invention. Detailed Implementation

[0036] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0037] The purpose of this invention is to overcome the shortcomings of existing technologies and propose an indoor robot visual navigation algorithm based on optimal policy playback. This algorithm consists of an on-policy algorithm and an off-policy algorithm. Catastrophic forgetting is reduced by training the current navigation target in the on-policy phase and reviewing old navigation targets in the off-policy phase. Unlike typical experience-based playback algorithms, this algorithm only stores optimal experience in memory to ensure that the agent can review the optimal policy of old targets while learning new targets. The algorithm trains a single feature extraction network and a policy network. The input to the policy network is the output of the state feature extraction network, i.e., the high-level feature map of the current image. The output of the entire network is two functions: a policy function π(a|s) and a value function Q(s|a). The network does not increase its parameters based on the number of targets and does not access any information about the environment or targets before training. The entire training process is divided into two phases: an on-policy phase and an off-policy phase. In the on-policy phase, the agent learns new policies for new goals through interaction with the environment, while in the off-policy phase, the agent reviews learned policies using best practices of old goals stored in memory to prevent catastrophic forgetting.

[0038] The technical solution adopted in this invention includes the following steps:

[0039] (1) Virtual environment preparation:

[0040] Our method was trained and tested in the AI2-THOR virtual environment. A navigation task in the AI2-THOR virtual environment includes a scene S, an initial point p, and a navigation target o. The agent's goal is to find the navigation target o in the 3D environment from the initial position within a given number of steps. The agent's action space is limited to six actions: forward, left turn, right turn, look down, look up, and Done. In each step, the robot receives a 300*300*3 image from the current camera and can act in one of two ways: 1. Select an action and move accordingly, or 2. Select the Done action to terminate the task. The set of all steps from the start to the end of the task is called an episode (trajectory). The task is considered successful if the following four conditions are met: 1. The robot selects the Done action; 2. The robot is no more than 1 meter away from the given target object; 3. The target object is within the robot's field of vision; 4. The robot does not exceed the maximum number of steps.

[0041] To reflect the sequential learning of navigation targets, ten navigation targets were set: "Alarm Clock," "Book," "Bowl," "Coffee Machine," "Kettle," "Plate," "Frying Pan," "Toaster," "Pot," and "Refrigerator." During training, one million trajectories were trained for each navigation target, in the order of the aforementioned navigation targets.

[0042] (2) Algorithm Flow Introduction

[0043] The algorithm process is divided into two parts: on-policy learning and off-policy learning. The details of each part are as follows:

[0044] 1) On-policy learning

[0045] In the on-policy phase, the goal is to train the agent to learn the optimal policy for the current target. When learning a new navigation policy, the agent interacts directly with the environment, and the policy gradient is given by the following formula:

[0046]

[0047] Where θ is the parameter of the neural network, γ∈[0,1) is the discount factor, and π θ (a t |s t This is the current strategy. It is an estimation of the action value function. It is an estimate of the value function, where k is the number of actions. Algorithm 1 shows the pseudocode for the on-policy process.

[0048] Algorithm 1:

[0049]

[0050] 2) Off-policy learning

[0051] Unlike typical off-policy algorithms, experience is not collected during the on-policy learning phase in the off-policy phase. Instead, experience is collected when the number of training trajectories for a navigation target reaches its maximum. This ensures that the collected experience represents the best experience for the current navigation target (assuming the task's success rate increases with training iterations), thus minimizing forgetting. Trajectories stored in memory are represented as episode = {(s...} t ,a t ,r t ,π t Q t )|1≤t≤T}, target=(target1,target2,…,target n ), where T is the total number of steps in the trajectory, and m is the memory size. The expression for memory is as follows:

[0052]

[0053] From a strategic perspective, memory stores:

[0054] Memory = (π1,π2,π3,…,π) n (8)

[0055] The off-policy phase is performed multiple times after the policy learning phase. The training process follows the off-policy algorithm of the ACER algorithm and uses Retrace Q. ret (x t ,a t To estimate Q π (a t |s t The algorithm uses bias-corrected truncation importance sampling to reduce the variance of the off-policy distribution drift. This algorithm successfully corrects the distribution shift corresponding to empirical replay. The policy gradient expression for the off-policy algorithm is as follows:

[0056]

[0057] in, It is the importance weight of the truncation. in [x] +=x, where c is a constant. Based on this loss, we added two more losses, namely L... policy-loss and L value-loss These two losses use KL divergence and L2 norm to fit the difference between the target policy and the behavioral policy, respectively. Algorithm 2 shows the pseudocode for policy learning. Algorithm 3 shows the pseudocode for the complete algorithm flow of this invention.

[0058] Algorithm 2:

[0059]

[0060] Algorithm 3:

[0061] Specific implementation examples:

[0063] (1) Preparation of virtual environment and navigation target:

[0064] First, download the AI2-THOR virtual environment package and set "Alarm Clock," "Book," "Bowl," "Coffee Machine," "Kettle," "Plate," "Frying Pan," "Toaster," "Pot," and "Refrigerator" as navigation targets. Before each training session, the robot is initially created anywhere in the room. One million trajectories are trained for each navigation target, in the order described above.

[0065] Prepare test cases. In an environment containing navigation targets, prepare 300 test cases for each navigation target. After training is completed, test the saved network parameters.

[0066] (2) Network Model

[0067] Network models such as Figure 1 As shown, the model is divided into a feature extraction module and a policy network module. The feature extraction module consists of an original image extraction module and an object detection module. For the original RGB image, the model uses a ResNet18 network pre-trained on ImageNet to extract features; simultaneously, it uses the feature information from object detection, as well as bounding boxes, confidence scores, and one-hot encodings (to determine whether it is a navigation target). The policy network module uses an LSTM network and two MLP networks as inputs to the policy function and Q function, and determines the action at the current time step based on the policy function.

[0068] (3) Evaluation indicators

[0069] The evaluation metric is the average success rate. This metric represents the average success rate across all trained navigation targets.

[0070] (4) Training process

[0071] At the start of a training trajectory, the environment randomly places the robot in an arbitrary location and assigns it the current navigation target as its task objective. At each moment, the robot receives the current image and selects the corresponding action based on it. After a trajectory is completed, the network undergoes an on-policy update based on the current trajectory. Following the on-policy update, several off-policy updates are performed to prevent the robot from forgetting old objectives.

[0072] Each navigation target will be learned one million times, and when a navigation target reaches its maximum number of times, it will be replaced with the next navigation target.

[0073] (5) Network Testing

[0074] For each saved model, its learned navigation goals are tested. During testing, each test case has a fixed starting position and navigation goal. The maximum step size during testing is 50 steps. After testing, the success rate is determined based on the number of successful attempts. A higher success rate indicates a lower degree of forgetting.

[0075] Table 1 shows the average success rate of this invention, where OPR (Optimal policy replay) is the method proposed in this invention, and xx K represents the maximum trajectory capacity of memory.

[0076] Table 1. Average success rate of this invention (see graph)

[0077] AlarmClock Book Bowl CoffeeMachine Kettle Plate Pan Toaster Pot Fridge OPR 50k 0.826 0.800 0.803 0.798 0.770 0.760 0.751 0.743 0.737 0.728 OPR 30K 0.825 0.793 0.774 0.739 0.711 0.667 0.679 0.655 0.671 0.660 OPR 5k 0.820 0.792 0.759 0.758 0.680 0.677 0.677 0.661 0.655 0.650 OPR 2k 0.800 0.777 0.705 0.697 0.639 0.622 0.627 0.621 0.617 0.592 A3C 0.823 0.458 0.309 0.263 0.230 0.172 0.243 0.196 0.181 0.117 EWC 0.825 0.475 0.519 0.324 0.125 0.164 0.08 0.06 0.07 0.07 global distribution 0.930 0.218 0.206 0.185 0.175 0.094 0.083 0.051 0.037 0.007

[0078] The results show that the method of the present invention achieves better results compared with other methods.

Claims

1. An indoor robot visual navigation method based on optimal strategy playback, characterized in that, Includes the following steps: Step 1: Virtual Environment Preparation In a virtual environment, a navigation task includes a scene S, an initial point p, and a navigation target o; the goal of the agent is to find the navigation target o in the 3D environment from the initial position within a given number of steps; the agent's action space is limited to six actions: move forward, turn left, turn right, look down, look up, and complete. In each step, the robot receives an image from the current camera in the scene and acts in one of two ways: 1) selects an action and moves accordingly, or 2) selects to complete the action and terminate the task; the set of all steps from the start to the end of the task is called a trajectory. The task is considered successful if all four of the following conditions are met: 1) The robot chooses to perform an action; 2) The robot is no more than 1 meter away from the given target object; 3) The target object is within the robot's field of vision; 4) The robot does not exceed the maximum number of steps. Multiple navigation targets were set up. During training, 1 million trajectories were trained for each navigation target, and the training order was the same as that of the navigation targets. Step 2: Robot visual navigation; Step 2-1: On-policy learning; In the on-policy phase, the goal is to train the agent to learn the optimal policy for the current target. When learning a new navigation policy, the agent interacts directly with the environment, and the policy gradient is given by the following formula: （1） in, These are the parameters of the neural network. This is the current strategy. It is an estimation of the action value function. It is an estimation of the value function. k It is the number of actions. It is the total number of steps in the trajectory. Is The action chosen by the agent at any given moment Is The current state of the agent; Step 2-2: Off-policy learning In the off-policy phase, experience is collected when the number of training trajectories for a navigation target reaches its maximum value; it is then stored... Memory The trajectory in is represented as , Where T is the total number of steps in the trajectory. m yes Memory size; Memory The expression is as follows: （2） in, Is The reward obtained by the intelligent agent at any given moment. Is The agent's strategy at any given moment Is The agent's Q-function at time t; from Memory From the perspective of preservation strategy, Memory The strategy for saving in is: （3） in, express The optimal strategy; Step 3: The off-policy phase is performed multiple times after the policy learning phase; Retrace is used. To estimate And bias-corrected truncation importance sampling is used to reduce the variance of off-policy distribution drift; where The formula is as follows: （4） The optimization objectives for off-policy are as follows: （5） in, It is the importance weight of the truncation. ,in , , c It is a constant. It is a strategy Expectations It is a discount factor; Based on the above optimization objectives, KL and L2 losses are further used to constrain the difference between the target policy and the behavioral policy.

2. The indoor robot visual navigation method based on optimal strategy playback according to claim 1, characterized in that, The virtual environment is the AI2-THOR virtual environment.

3. The indoor robot visual navigation method based on optimal strategy playback according to claim 1, characterized in that, The robot receives an image from the current camera in the scene, and the image size is 300*300*3.

4. The indoor robot visual navigation method based on optimal strategy playback according to claim 1, characterized in that, The navigation targets are set to 10: "alarm clock", "book", "bowl", "coffee machine", "kettle", "plate", "frying pan", "toaster", "pot", and "refrigerator".

Images