Vision navigation method based on autonomous learning

Abstract

The application discloses a kind of visual navigation methods based on autonomous learning, belong to navigation technical field.The application constructs the visual navigation model based on deep reinforcement learning, it is input into network with the target of agent as input, so that model has universality, the image of current observation and target is handled by ResNet-50, and the extracted features are input into A3C network, output the next action of agent and the evaluation of strategy, after continuously optimizing strategy network and value network, the visual navigation model with stable output performance is obtained.In order to stabilize the optimal value of approaching loss function, the learning rate decreases exponentially in a ladder shape, and finally stabilizes at a very small learning rate, avoiding the model oscillating around the optimal value of loss function.Through the optimization of learning rate algorithm, the training efficiency of visual navigation model is higher, and the loss function converges faster.

Description

Technical Field

[0001] This invention belongs to the field of navigation technology, specifically relating to a visual navigation method based on autonomous learning. Background Technology

[0002] Deep reinforcement learning is an end-to-end perception and control system with strong versatility. The Asynchronous Advantage Actor-Critic (A3C) algorithm is a lightweight deep reinforcement learning framework that performs best on various control tasks in continuous action spaces. Deep reinforcement learning can perceive the environment and make decisions; when applied to visual navigation, agents can interact with the environment to obtain information, thereby navigating and learning.

[0003] In visual navigation, the agent acquires the current observations and passes them to the network, outputting an action probability distribution. Actions are obtained by random sampling. After the agent applies the actions to the current environment, it acquires new observations again, which is the state transition of the environment.

[0004] In traditional deep reinforcement learning, the agent iterates through the environment based on current observations until it reaches the target location. In such a model, the policy function is learned only by the current state, while the navigation target needs to be implicitly embedded into the model's network parameters. Therefore, when the navigation target changes, new navigation parameters need to be trained, resulting in poor generalization ability. Summary of the Invention

[0005] This invention provides a visual navigation method based on autonomous learning. By optimizing the learning rate algorithm of the visual navigation model during the training process, an optimized visual navigation model is constructed to improve navigation performance.

[0006] The technical solution adopted in this invention is a visual navigation method based on autonomous learning, which includes the following steps:

[0007] Step 1: Configure the action space of the visual navigation agent in the virtual environment, and set the reward strategy for the agent;

[0008] The input data of the intelligent agent includes currently observed image data and target image data. The currently observed image data refers to video image frames acquired by the intelligent agent at its current location; the target image data refers to video image frames at the target location of the intelligent agent.

[0009] Step 2, construct a visual navigation model;

[0010] The visual navigation model includes a general twin layer and a scene-specific layer;

[0011] Among them, the general twin layer uses two weight-sharing twin networks to transform the current observation and the target into the same embedding space for perceiving the scene layout, and then inputs the perception results of the scene layout into the corresponding scene of the scene-specific layer to obtain the policy distribution and policy value.

[0012] The twin network includes a residual network ResNet-50, a fully connected layer, and a fusion layer. The residual network ResNet-50 is used to extract feature information from the image data of the current observation and the target. The feature information of the current observation and the target image is concatenated through the fully connected layer, and then a joint feature is output through the fusion layer for processing at a scene-specific layer.

[0013] The scenario-specific layer includes several scenario networks, each corresponding to a scenario. The scenario network includes a policy network and a value network. The policy network selects matching actions based on probability distribution, and the value network evaluates and scores the currently selected actions to obtain a score. The policy network then modifies itself based on the current score, so that the two networks complement each other.

[0014] Step 3: Train the visual navigation model to obtain a visual navigation model for the target agent;

[0015] During training, the root mean square propagation optimizer was used to optimize the network parameters of the visual navigation model;

[0016] First, initialize the learning rate η as follows:

[0017]

[0018] Where, η initial This represents the initial learning rate, α is the preset adjustment factor, and e is the natural base.

[0019] Furthermore, during the learning rate warm-up phase, the learning rate η during the t-th iteration of training is... t Updated to:

[0020]

[0021] Among them, T warmup This indicates the number of iterations during the learning rate warm-up phase.

[0022] After the learning rate warm-up phase, the learning rate η is set to:

[0023]

[0024] Where β represents the decay exponent, step represents the time step, and T max denoted by , where k represents the total number of iterations, and k represents the hyperparameter controlling how the learning rate changes.

[0025] The technical solution provided by this invention brings at least the following beneficial effects:

[0026] (1) This invention constructs a visual navigation model based on deep reinforcement learning. The core of this model is to input the agent's target as input into the network to make the model universal. The current observation and target image (e.g., RGB image) are processed by ResNet-50, and the extracted features are input into the A3C network to output the agent's next action and policy evaluation. After continuous optimization of the policy network and value network, a visual navigation model with stable performance is output.

[0027] (2) The learning rate optimization method employed starts the model with a very low learning rate during network parameter initialization to avoid learning too much erroneous information. The learning rate is then continuously increased, maintaining a steady increase for a period until it reaches its maximum value. To stabilize and approach the optimal value of the loss function, the learning rate decreases exponentially in a step-like manner, eventually stabilizing at a very small learning rate to prevent the model from oscillating around the optimal value of the loss function. This optimization of the learning rate algorithm results in higher training efficiency for the visual navigation model and faster convergence of the loss function. Attached Figure Description

[0028] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0029] Figure 1 This is an embodiment of the invention, showing the bathroom observation RGB image within the AI2THOR framework;

[0030] Figure 2 This is a schematic diagram of the network architecture of the visual navigation model provided in an embodiment of the present invention;

[0031] Figure 3 In this embodiment of the invention, the expression tree is graphically represented. Detailed Implementation

[0032] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.

[0033] To avoid retraining when changing navigation targets and to achieve high generalization ability, a target-driven navigation model is used. The network input of this model is the agent's current observations and the visualized target location. New targets can be flexibly specified when using the model, so the navigation target no longer needs to be trained into the navigation parameters. Since the training target and the new target have a certain correlation, the trained model learns the new target more easily.

[0034] Actor-Critic (AC) is a reinforcement learning method. It consists of two parts: (1) Policy Network Actor: approximates the policy function π(a|s) using a neural network π(a|s; θ); (2) Value Network Critic: approximates the action value function Q using a neural network q(s,a;w). π (s,a). Where a represents the action, s represents the state, θ represents the parameters in the policy network, and w represents the parameters in the value network. The two parameters are mainly optimized after training so that θ can select the optimal action and w can accurately estimate the value.

[0035] Compared to Q-learning, which can only solve problems in discrete action spaces, the policy network selects appropriate actions based on probability distributions, the value network evaluates the action based on the evaluation score, and the policy network then modifies the policy network based on the evaluation score. The two networks complement each other.

[0036] The parameter update method for the Actor in AC using the TD algorithm as the policy gradient algorithm is as follows:

[0037]

[0038] q t ≈Q π (s t ,a t )

[0039]

[0040] Critic updates its parameters as follows:

[0041]

[0042]

[0043]

[0044] Where, θ t+1 θ t s represents the parameters in the policy network for the current and next iterations, respectively. t ,a tThis represents the current (t-th) state and action, α and β represent the learning rates of the parameters in Critic and Actor, and Q... π () represents the action value function, δ t y represents the current TD error. t This indicates the current target value.

[0045] The implementation of the model in the visual navigation process involves three key elements: action space, input data, and reward design.

[0046] (1) Action Space: The main considerations are four simple actions: forward, backward, left turn, and right turn. For forward and backward, a fixed step size of 0.5m is used in the AI2THOR virtual environment; for left and right turns, the turning angle is set to 90°. The virtual environment space is discretized into a grid world through these four actions.

[0047] (2) Input data: The input consists of a 224×224×3 RGB image of the agent's current observation in the virtual environment and a 224×224×3 RGB image of the target location. The expectation is not only that the agent navigates from the current observation to the target location, but also that it turns in the correct direction at the target location.

[0048] (3) Reward Design: After multiple training iterations, the desired model is one that enables the agent to navigate to the target observation from the current observation using the shortest possible trajectory. In a given space, a reward of 10 is set for reaching the final target location. Simultaneously, to incentivize the agent to take the shortest path, a penalty of -0.01 is imposed as a time penalty for each step taken by the agent that fails to reach the target. The agent may collide with unexplored areas in the environment, which is obviously a worse situation because the shortest path from the current observation to the target location will never result in a collision. Therefore, the time penalty for a collision is larger and set to -0.1.

[0049] Since visual navigation is based on deep reinforcement learning, it requires the agent to interact with the environment to obtain state information, i.e., the RGB image observed by the agent. Therefore, this embodiment introduces a virtual 3D environment. The development of the graphics community has enabled virtual 3D scenes to provide richer representations of the appearance, physical properties, and states of the real world, narrowing the gap between simulated scenes and the real world. AI2THOR simulates real-world scenes as closely as possible; it is a multi-functional architecture with high scalability, allowing different types of virtual scenes to be easily combined. The AI2THOR framework integrates the physics engine Unity 3D and the deep learning framework Tensorflow, allowing direct communication between the two. This embodiment uses the open-source scene file thor_v1_scene_dumps from the Stanford Vision and Learning Lab. The file contains four scenes: bathroom, bedroom, living room, and kitchen. These scenes and agents can be imported into the h5py library. After importing the bathroom scene file, the observation image obtained under the AI2THOR framework by controlling the agent's direction is shown below. Figure 1 As shown.

[0050] The core of visual navigation lies in learning the policy function π through deep reinforcement learning. To this end, a deep neural network is used to approximate the nonlinear function of the policy function π, i.e., at time t:

[0051] a~π(s t ,g|θ)

[0052] Where 'a' is the action generated according to the strategy, and 's' is the action generated according to the strategy. t θ is the RGB image currently observed by the agent in the virtual environment, θ is the parameter of the deep neural network, and g is the RGB image of the navigation target.

[0053] The model consists of two main modules: a general twin layer and a scene-specific layer, with the overall architecture as follows: Figure 2 As shown.

[0054] Image observation learns features through a general twin layer, and then inputs these features into a scene-specific layer to obtain policy distribution and policy value.

[0055] The general twin layer uses two weight-shared twin networks to transform the current observation and the target into the same embedding space for perceiving scene layout.

[0056] The general twin layer consists of three parts: the residual network ResNet-50, the fully connected layer (fc), and the fusion layer.

[0057] Using a ResNet-50 residual network (a 50-layer residual network) pre-trained on the ImageNet dataset, the input 224×224×3 RGB image is mapped to 2048-dimensional features.

[0058] The residual network introduces a residual module f i (x) and the connection layer that skips residual modules constitute a layer that allows data to flow from any layer to subsequent layers. (y) i-1 If the input is , then the expression for the output function of the i-th residual module is:

[0059] y i =f i (y i-1 )+y i-1

[0060] residual module f i (x) consists of a convolutional layer, a batch normalized layer, and a ReLU (·) activation function, i.e.:

[0061] f i (x)=w i *ReLU(B(w′ i *ReLU(B(x))))

[0062] Among them, w i ,w′ i denoted as the weight matrix, * denotes the convolution operation, and B(·) denotes the batch normalization operation.

[0063] For a residual network containing 3 residual modules, its output expression is:

[0064] y3=y0+f1(y0)+f2(y0+f1(y0))+f3(y0+f1(y0)+f2(y0+f1(y0)))

[0065] The above graphical representation of the residual network containing three residual modules is as follows: Figure 3 As shown in the diagram, there are many flow paths from input to output, and each flow path is a unique configuration module that can be skipped or not. Therefore, each module f in the residual network... i The data for (·) is 2. i-1 A mixture of different distributions, each derived from the possible configurations of the first i-1 residual modules.

[0066] Since residual networks can be viewed as a collection of many paths rather than a single deep network, and the paths are not strongly dependent on each other, they can effectively avoid the degradation phenomenon of deep networks and the problems of vanishing and exploding gradients. Residual networks with frozen parameters after pre-training on the ImageNet dataset can map the input RGB images in visual navigation models into better features.

[0067] Based on the 2048-dimensional features output by ResNet-50, the model concatenates the features from four historical frames to form an 8192-dimensional feature, which is used to explain the agent's historical actions. The features of the current observation and the target are mapped to the same embedding space, preserving the spatial geometric relationship between the current and target positions. The features of the current observation and the target are concatenated after passing through a fully connected layer, and a joint feature is output through a fusion layer, facilitating processing at scene-specific layers.

[0068] The scenario-specific layer is built based on the A3C algorithm and can be performed asynchronously and in parallel. After passing through the policy network and policy value network of the scenario-specific layer, the probability distribution and policy value of the four actions are output.

[0069] In this embodiment of the invention, for each scenario, there is a corresponding network at a specific layer of the scenario, and the parameters between the networks are not shared.

[0070] In deep learning algorithms, the network structure and optimization of neural networks are two crucial factors affecting algorithm performance. The optimization method of a neural network directly impacts the algorithm's speed and performance; therefore, selecting a suitable optimizer algorithm is essential. Neural network optimization refers to optimizing the neural network parameters during model training to ensure the loss function converges to its minimum or optimal value as quickly as possible.

[0071] Among current deep learning models, well-known optimizers include stochastic gradient descent, stochastic gradient descent with momentum, adaptive gradient descent, root mean square propagation (RMSprop), and adaptive moment estimation. In this embodiment of the invention, the RMSprop optimizer is used to optimize the parameters. The optimization method is as follows:

[0072]

[0073]

[0074]

[0075] Among them, v t Let θ represent the momentum at time t. tLet θ represent the parameters optimized at time t, η represent the learning rate, θ0 represent the initial optimized parameters, Loss represent the loss function, and α represent the decay factor.

[0076] As shown in the optimization formula above, the closer the gradient is to the current iteration number, the smaller its impact on the current gradient calculation. Furthermore, the learning rate is iteratively decayed using the iteration number and accumulated gradient.

[0077] Compared to stochastic gradient descent, the RMSprop optimizer adds momentum, which can be used to converge when the gradient is small, solving the problem of large amplitude during the update of the optimization function, and can also speed up the convergence of the loss function. Compared to adaptive gradient descent, RMSprop corrects its shortcoming of causing training to end prematurely.

[0078] The algorithm optimization is based on a visual navigation model; therefore, this embodiment of the invention incorporates an initial learning rate decay function into the model. As shown in the above equation, the RMSprop optimizer has already decayed the learning rate, so only an initial fixed learning rate needs to be input.

[0079] The initial learning rate is processed by a function in this embodiment so that it changes as the model trains before being input into the RMSprop optimizer.

[0080] The learning rate is initialized as follows:

[0081]

[0082] Where α is the adjustment factor and e is the natural base.

[0083] During the processing, warmup and annealing were also performed on the learning rate.

[0084] When training a neural network, RMSprop's learning rate decay uses a large initial learning rate that then decreases as the number of training iterations increases. This allows the model to optimize network parameters with a high learning rate initially, and then decreases the learning rate as the network model matures to stabilize it.

[0085] However, in the initial stage of training, due to the random initialization of weights, the initial model differs greatly from the expected model, and the expected gradient near the output layer is large. If a large learning rate is used at this time, the network will optimize the network parameters with non-standard evaluation, thus causing the optimization process to oscillate.

[0086] Using a warmup learning rate allows for a smaller learning rate to be used in the initial iterations, which slows down the model's optimization. After the maximum warmup time is reached, the model becomes relatively stable, at which point a larger learning rate is chosen to begin learning—the initial learning rate—which accelerates the model's convergence. In subsequent iterations, the learning rate then decays.

[0087] Warmup can be viewed as a function of the learning rate η with respect to the number of iterations t:

[0088]

[0089] Where t≤T warmup T warmup This indicates the number of iterations during the learning rate warm-up phase.

[0090] During the warmup phase, the learning rate starts from a small value and increases in some way, over a period of time (T). warmup The maximum η is reached after the second iteration. max .

[0091] After warmup, the learning rate is kept at a relatively large value to ensure fast convergence. Then, to avoid oscillations when the loss function converges to near its optimum, learning rate decay, or learning rate annealing, is used.

[0092] Common decay algorithms include piecewise constant decay, inverse time decay, exponential decay, natural exponential decay, and cosine decay.

[0093] In this embodiment of the invention, a relatively large value is set for the final learning rate of the warmup, which is the initial learning rate η. initial The learning rate is expected to decay at a relatively fast rate initially to ensure the model maintains a high learning rate in the short term; subsequently, a slower decay rate is used to ensure the model converges slowly near the optimum. Therefore, a combination of piecewise constant decay and exponential decay is used, resulting in a piecewise exponential decay of the learning rate initially, followed by piecewise decay:

[0094]

[0095] Where β represents the decay exponent, step represents the time step, and T max denoted by , where k represents the total number of iterations, and k represents the hyperparameter controlling how the learning rate changes.

[0096] In the training process of the visual navigation model, warmup and annealing were added by optimizing the learning rate decay, in order to enable the model to converge faster and better.

[0097] To further verify the navigation performance of the self-learning-based visual navigation method provided in this embodiment of the invention, a visual navigation experiment was conducted in a virtual 3D scene. The scene bathroom from the open-source scene file thor_v1_scene_dumps of the Stanford Vision and Learning Lab was imported into the h5py library to retrieve the experimental scene and agent.

[0098] The established visual navigation model was applied to the bathroom scene, with five target locations set as numbered 26, 37, 43, 53, and 69. Twenty threads were used to navigate these tasks in parallel to accelerate the training process.

[0099] After training the two models for a sufficient period of time, the trained models were tested, and the test data was output.

[0100] For the test in the bathroom scenario, the target used during training was first selected. Each navigation test was performed 100 times, and the average value was output. A total of 10 tests were run.

[0101] The data output results are as follows: the test data of the original model is shown in Table 1, and the test data of the optimized model is shown in Table 2.

[0102] Table 1. Test trajectory length of the original model in Bathroom

[0103]

[0104] Table 2. Test trajectory length of the optimized model in Bathroom.

[0105]

[0106]

[0107] Comparing the two sets of data, the difference in trajectory length is not significant, and neither model is definitively superior. In other words, once the models are well-trained, their performance is not significantly different.

[0108] Furthermore, this embodiment also selects targets not present in the training data for generalization testing. Since the model has poor navigation capabilities for new targets, the output data is not the trajectory length but the navigation success rate.

[0109] With other parameters unchanged, the navigation success rates of the original model are shown in Table 3. The success rates for targets 52 and 68 are both 100%, so there is not much comparative value. The success rates for targets 25 and 42 are almost the same, indicating that after the models are trained and mature, there is not much difference in the navigation performance of the two models. However, the navigation success rates of both are not high, and their generalization ability is relatively weak.

[0110] Table 3 Generalization test results data

[0111]

[0112] This embodiment records the trajectory length, maximum policy value, and cumulative reward at each iteration during navigation training. Trajectory length represents the number of iterations from the starting point to the target location, providing a direct indication of navigation performance; a shorter trajectory length indicates better navigation. Maximum policy value represents the output of the currently constructed value network. As the model trains and the Critic's evaluation ability improves, this value becomes more reliable. Cumulative reward represents the sum of all rewards and penalties obtained after a successful navigation iteration. A larger cumulative reward indicates that the navigation reaches the target faster and minimizes collisions with unreachable areas in the selected strategy. Observing these three values ​​allows for a comprehensive assessment of navigation performance changes during training, with trajectory length being the most intuitive indicator.

[0113] In this embodiment, although both the optimized and unoptimized models eventually converge to the same shortest trajectory length, the optimized model clearly exhibits better convergence. Compared to the original model, the optimized model can be trained to maturity in a shorter time, or, with the same training time, is more reliable. Furthermore, both models, upon final optimization, show the same value for each policy choice, indicating a high degree of similarity between their policy and value networks. The optimized model demonstrates higher policy value for policies chosen before model maturity, indicating more accurate policy evaluation or superior policy selection, suggesting better optimization of both the policy and value networks. Considering that the optimized model almost always yields better cumulative rewards for each navigation—meaning that the trajectory is almost always shorter than that of the unoptimized model—this strongly supports the superiority of the optimized model.

[0114] Since the significant change in training performance before and after optimization occurs within the range of training frames [0, 1M], analysis and evaluation of the first 1M training frames revealed that: because the average trajectory length of the model's training performance is independent of the initial location of the input model, the initial average trajectory length for different target locations is related to the initial weights. After initializing the model's network parameters, for a certain set of target locations, such as target 69, the network parameters initialized before optimization provide better navigation for that target. However, for targets 37, 43, and 53, the network parameters initialized after optimization provide better navigation for these targets.

[0115] Define a physical quantity—the training rate—to quantitatively evaluate the model's performance:

[0116]

[0117] Where T is the evaluation interval, Δlength is the average trajectory length difference per K training frames of the selected 100K training frames, and Δt K The selected 100K training frames. The training rate refers to the average reduction in trajectory length per K frames per 100K training frames within the evaluation interval, and the maximum value within the evaluation interval is taken as the training rate of the model.

[0118] Obviously, for two models with similar performance after maturation, the evaluation interval set in this embodiment is within the training interval before the models mature. Within this interval, the training rate of the two models is calculated. Since the average trajectory length decreases smoothly, the learning rate can be used to evaluate the overall rate of decrease of the average trajectory length within the training interval, i.e., the model performance.

[0119] Furthermore, regardless of the quality of the initial network parameters, the optimized model consistently maintains a performance superior to the unoptimized model at this stage:

[0120] (1) The network initialization parameters do not significantly affect the navigation performance of the target. For example, for target 26: during model training, the average trajectory length curve of the optimized model decreases faster than that of the unoptimized model. When 1M data frames are set as the evaluation interval, the highest training rate of the unoptimized model is calculated to be 1.1952, and the highest training rate of the optimized model is 1.8039.

[0121] (2) The navigation effect of the network initialization parameters of the model before and after optimization is different for the target. For example, the navigation effect of the model before optimization is better for target 69, while the navigation effect of the model after optimization is better for targets 37, 43, and 53. For these targets, the network models are qualitatively compared. During the training process of 1M data frames, if the initial navigation effect of the optimized model is good, it will maintain this excellent navigation effect, and the average trajectory length will always be smaller than that of the model before optimization. However, if the initial navigation effect of the model before optimization is good, the optimized model will maintain a higher training rate of 5.2578 when the training rate of the model before optimization is 0.4137, so that the average trajectory length of navigation can quickly catch up with that of the model before optimization, and then maintain this excellent navigation effect, so that the average trajectory length is smaller than that of the model before optimization.

[0122] In this implementation, the original model and the optimized model were applied to a virtual 3D environment under AI2THOR. Through training and testing, the differences between the original model and the optimized model were compared. The average trajectory reduction within 1K training frames was defined as the training rate, and the navigation performance was evaluated in combination with the training data graph.

[0123] This invention constructs a visual navigation model based on deep reinforcement learning, enabling an intelligent agent to build decision and value networks through autonomous exploration. By using AI2THOR to build a virtual 3D scene, the visual navigation model is applied to this scene, training each target task asynchronously and in parallel, outputting a model usable for navigation, including a pre-trained policy and value network.

[0124] This invention studies learning rate algorithms. Based on a survey of various learning rate algorithms, it optimizes the learning rate algorithm by combining warmup and annealing decay as optimization methods. The optimized learning rate algorithm is then incorporated into a navigation model to construct an optimized visual navigation model. Both the unoptimized and optimized models are trained on the same scale using the same training task, and navigation tests are performed on both models after training. The output data during training and the test output data after training are compared. By defining a new physical quantity, training rate, the superiority of the optimized algorithm in model training is analyzed.

[0125] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

[0126] The above descriptions are merely some embodiments of the present invention. Those skilled in the art can make various modifications and improvements without departing from the inventive concept of the present invention, and these all fall within the scope of protection of the present invention.

Claims

1. A visual navigation method based on autonomous learning, characterized in that, Includes the following steps: Step 1: Configure the action space of the visual navigation agent in the virtual environment, and set the reward strategy for the agent; The input data of the intelligent agent includes currently observed image data and target image data. The currently observed image data refers to video image frames acquired by the intelligent agent at its current location; the target image data refers to video image frames at the target location of the intelligent agent. Step 2, construct a visual navigation model; The visual navigation model includes a general twin layer and a scene-specific layer; Among them, the general twin layer uses two weight-shared twin networks to transform the current observation and target into the same embedding space for perceiving the scene layout, and then inputs the perception results of the scene layout into the corresponding scene of the scene-specific layer to obtain the policy distribution and policy value. The twin network includes a residual network ResNet-50, a fully connected layer, and a fusion layer. The residual network ResNet-50 is used to extract feature information from the image data of the current observation and the target. The feature information of the current observation and the target image is concatenated through the fully connected layer, and then a joint feature is output through the fusion layer and sent to the scene-specific layer. The scenario-specific layer includes several scenario networks, each corresponding to a scenario. The scenario network includes a policy network and a value network. The policy network selects matching actions based on probability distribution, and the value network evaluates and scores the currently selected actions to obtain a score result. The policy network then modifies the policy network based on the current score. Step 3: Train the visual navigation model to obtain a visual navigation model for the target agent; During training, the root mean square propagation optimizer was used to optimize the network parameters of the visual navigation model; First, initialize the learning rate η as follows: Where, η initial This represents the initial learning rate, α is the adjustment factor, and e is the natural base. Furthermore, during the learning rate warm-up phase, the learning rate η during the t-th iteration of training is... t Updated to: Among them, T warmup This indicates the number of iterations during the learning rate warm-up phase. After the learning rate warm-up phase, the learning rate η is set to: Where β represents the decay exponent, step represents the time step, and T max denoted by , where k represents the total number of iterations, and k represents the hyperparameter controlling how the learning rate changes.

Images