Visual navigation method and system based on goal-driven and intrinsic motivation exploration

Abstract

The application belongs to the technical field of robot vision navigation, and particularly discloses a vision navigation method and system based on target driving and internal motivation exploration, which comprises the following steps: S1, generating an offline data set; S2, inputting an observation image and a target image at a current moment into a feature extraction module to extract a feature vector at the current moment; S3, inputting the feature vector into an A3C network to predict an optimal navigation action, and interacting with the environment to obtain an observation image at a next moment and an external reward; S4, inputting the observation image, the target image and the feature vector in the step S2 into an internal reward calculation module to obtain an internal reward, combining the external reward to calculate a loss value, and updating network parameters of the feature extraction module and a deep reinforcement learning model. By predicting a depth map and a semantic segmentation map of the observation image and the target image, the navigation ability in a simulated indoor environment is improved, the problem of sparse reward in navigation is solved, and the convergence speed of the model is improved.

Description

Technical Field

[0001] This invention belongs to the field of robot visual navigation technology, and relates to a visual navigation method and system based on goal-driven and intrinsic motivation exploration. Background Technology

[0002] Visual navigation is a task in which an agent (such as a mobile robot in an environment) can navigate to a destination using only visual input. Goal-driven visual navigation, on the other hand, is to give the agent a target image (the image that the agent sees at the target location) and its goal is to move from the current position to the target position using the shortest path through a series of actions (e.g., forward, rotation, backward, etc.) based solely on the RGB high-dimensional image observed by a monocular camera in the environment.

[0003] Traditional methods primarily rely on maps, meaning a map is built first and then planning is performed, encompassing localization, mapping (SLAM), and path planning. However, this method is unusable in unknown environments, cannot meet the diverse needs of real-time scenarios, and the hardware costs required for localization and mapping are also high.

[0004] Currently, visual navigation based on deep reinforcement learning has been proven to outperform traditional methods. This approach directly uses visually observed images as input to predict navigation actions. First, image features are extracted using a deep neural network, and then these features are input into a policy network within the reinforcement learning process to predict navigation actions.

[0005] However, many existing technologies require pre-trained feature extractors, such as the DETR object detector and ResNet network, to extract useful environmental features. Some methods rely on external knowledge bases to construct semantic relationships between objects to extract semantic features, which cannot meet the requirements of a fully end-to-end solution. There are also some methods that directly apply reinforcement learning frameworks to visual navigation, but due to the high-dimensional image input and the problem of reward sparsity, the model training time is long, the convergence is poor, and the generalization ability is weak.

[0006] Furthermore, problem exploration remains one of the major challenges in reinforcement learning. The goal of the classic reinforcement learning paradigm is to maximize external rewards. However, due to the partial observability of the visual navigation environment, the agent receives no external rewards until it reaches the target. This makes it difficult for an agent lacking initial knowledge of its environment to collect meaningful reward signals for learning. The agent may end up operating in only certain areas of the environment, failing to accurately explore unknown areas to find the target, thus prolonging navigation time. Secondly, if the agent in the navigation environment cannot access each state-action pair an unlimited number of times, it will prematurely converge to a local optimum after a finite number of steps, failing to learn the shortest path to the target, which affects navigation accuracy.

[0007] Therefore, in such complex, reward-sparse high-fidelity simulation environments, how to extract useful image features and how to learn an effective exploration strategy is an urgent problem to be solved. Summary of the Invention

[0008] The purpose of this invention is to provide a visual navigation method and system based on goal-driven and intrinsic motivation exploration, which improves navigation capabilities in simulated indoor environments and solves the reward sparsity problem in navigation.

[0009] To achieve the above objectives, the basic solution of this invention is: a visual navigation method based on goal-driven and intrinsic motivation exploration, comprising the following steps:

[0010] S1 simulates a mobile robot traversing every state in the AI2-THOR simulation environment, generating an offline dataset for the AI2-THOR simulation platform.

[0011] S2, input the observed image and target image at the current moment into the feature extraction module to extract the feature vector at the current moment;

[0012] S3. Input the feature vector obtained in step S2 into a deep reinforcement learning model (such as A3C network model, PPO model, etc.) to predict the optimal navigation action and interact with the environment to obtain the observation image and external reward at the next moment.

[0013] S4. Input the observed image, the target image, and the feature vector from step S2 into the intrinsic reward calculation module to obtain the intrinsic reward. Combine the intrinsic reward with the extrinsic reward to calculate the loss value. Use the loss value to update the network parameters of the feature extraction module and the deep reinforcement learning model.

[0014] The working principle and beneficial effects of this basic scheme are as follows: This technical scheme enables the feature extractor to contain rich depth and semantic information by predicting the depth map and semantic segmentation map of the observed and target images, thereby accelerating the convergence speed of the network. It utilizes intrinsic reward signals to encourage the agent to explore unknown areas in the environment, dividing these rewards into long-term and short-term rewards, thus learning a better exploration strategy, accelerating the learning speed, and enabling the agent to reach the target in a shorter time.

[0015] Furthermore, the method for generating the offline dataset for the AI2-THOR simulation platform in step S1 is as follows:

[0016] S11, Download the Python file to construct the offline environment, set the cell size after meshing of the AI2-THOR simulation environment and the image size parameters observed at each coordinate, take multiple scenes of different spatial types, use all scenes as the training set, and take one scene from each space as the test set;

[0017] S12 simulates a mobile robot traversing each coordinate in the AI2-THOR simulation environment, simultaneously acquiring the corresponding RGB image, RGB-D depth image, and RGB-S semantic segmentation image, saving the transformation relationship between each coordinate, saving the image information corresponding to the coordinate, and generating an offline dataset.

[0018] Navigation is performed in various indoor environments, generating offline datasets for future use.

[0019] Furthermore, the method for extracting the feature vector at the current time step S2 is as follows:

[0020] S21, view the current observation image s t and target image s target The two feature vectors are input into the Siamese convolutional neural network respectively to obtain their feature vectors. The two feature vectors are then fused using convolutional network operations to obtain the fused feature vector of the observed image and the target image.

[0021] S22, the fused feature vector is input into three deconvolutional neural networks to predict the depth map of the observed image. Semantic segmentation graph semantic segmentation map of the target image Then, the PyTorch built-in 2D average pooling function `avg_pool2d` is used to perform average pooling on the depth map and semantic segmentation map in the offline dataset to obtain the ground truth value `d`, which is the same size as the predicted map. s s s s target Then, the total loss value L of the prediction result is calculated using the mean square error MSELoss. deconv The specific formula is as follows;

[0022]

[0023] S23. The fused features obtained in step S21 are concatenated with the action distribution features and normalized reward features from the previous time step, and then input into the Long Short-Term Memory network along with the hidden state features from the previous time step to extract feature vectors containing historical features.

[0024] Among them, the action distribution feature indicates which action was selected in the previous moment. Taking the four actions of forward, backward, left turn, and right turn as an example, if the action of "forward" was selected in the previous moment, the action distribution feature is a 4×1 vector [1,0,0,0]. The reward feature is a 1×1 vector between [0-1] after normalizing the external reward obtained in the previous moment. The hidden state feature is an output of the long short-term memory network in the previous moment, representing the historical state feature. In this invention, the size of this vector is 512×1.

[0025] By predicting the depth map and semantic segmentation map of the observed and target images, the feature extractor contains rich depth and semantic information, which speeds up the convergence speed of the network.

[0026] Furthermore, the Siamese convolutional neural network includes a first two-dimensional convolutional layer containing 32 filters, a 7×7 convolutional kernel, a span of 4, and an activation function of ReLU, and a second two-dimensional convolutional layer containing 32 filters, a 4×4 convolutional kernel, a span of 2, and an activation function of ReLU.

[0027] The current observation image s t and target image s target The first two-dimensional convolutional layer and the second two-dimensional convolutional layer are input sequentially to obtain the observed image s. t and target image s target eigenvectors;

[0028] Use feature concatenation operations (such as the cat and stack functions) to concatenate the observed image s. t and target image s target The feature vectors are concatenated and then sequentially input into a two-dimensional convolutional layer containing 64 filters, a 4×4 convolutional kernel, a span of 2, and a ReLU activation function, and a two-dimensional convolutional layer containing 32 filters, a 1×1 convolutional kernel, a span of 1, and a ReLU activation function, to obtain the fused feature vector of the observed image and the target image.

[0029] A twin convolutional neural network structure is set up to extract and fuse the feature vectors of the observed image and the target image to extract feature information.

[0030] Furthermore, the three deconvolutional neural networks include a third two-dimensional deconvolutional layer containing 16 filters, a 4×4 convolutional kernel, a span of 2, and a ReLU activation function; a fourth two-dimensional deconvolutional layer containing 1 filter, a 4×4 convolutional kernel, and a span of 2; a fifth two-dimensional deconvolutional layer containing 16 filters, a 4×4 convolutional kernel, a span of 2, and a ReLU activation function; and a sixth two-dimensional deconvolutional layer containing 3 filters, a 4×4 convolutional kernel, and a span of 2.

[0031] The fused feature vectors are passed through the third and fourth two-dimensional deconvolution layers to predict the depth map of the observed image.

[0032] The fused feature vectors are passed through the fifth and sixth two-dimensional deconvolution layers to predict semantic segmentation maps of the observed and target images.

[0033] By setting up a deconvolutional neural network structure, depth maps and semantic segmentation maps are extracted from the fused feature vectors, thereby accelerating the convergence speed of the network.

[0034] Furthermore, the method for obtaining the observation image and external reward at the next time step in step S3 is as follows:

[0035] S31, Input the fused feature vector obtained in step S2 into the A3C network to predict the agent's optimal navigation action at that moment;

[0036] S32, the agent executes the optimal navigation action, using the offline dataset to move to the coordinates of the next time step and obtain the corresponding observation image s. t+1 Then, based on the reward function, calculate the external reward at that moment.

[0037]

[0038] Using the observed and target images as input, the system directly outputs the predicted optimal navigation action and calculates external rewards for subsequent use.

[0039] Furthermore, the A3C network includes a policy network with 4 hidden units and a value network with 1 hidden unit (the number of selectable navigation actions corresponds to the number of hidden units in the policy network);

[0040] The fused feature vectors are input into the policy network and the value network respectively. The value network outputs the value at that moment. A softmax operation is performed on the output of the policy network to obtain the probability of each action being selected. The action with the highest probability is selected as the optimal navigation action at that moment.

[0041] Using visually observed images and target images as input, the system directly predicts the optimal action for each step to complete the visual navigation task.

[0042] Furthermore, in the intrinsic reward calculation module of step S4, the intrinsic reward is divided into long-term and short-term intrinsic rewards. This prevents the agent from accessing the same state in the short term and gradually prevents access to the same state in the long term. Specifically, this includes the following steps:

[0043] S41, input the fused feature vector obtained in step S2 into the short-term reward calculation module to obtain the short-term intrinsic reward within one iteration. The short-term reward calculation module adopts a counting method based on state abstraction, using a memory M to store the historical abstract state in one iteration. At time t, the stored information in M ​​is {f(s1), f(s2)...f(s...}}. t-1 )}, where f represents the feature extraction function, s t-1 For observation images;

[0044] By compressing the original state space, a pseudo-count is performed on the abstract state of the currently observed image, and the reciprocal of the pseudo-count is used as the short-term intrinsic reward.

[0045]

[0046]

[0047] Wherein, nf(s) t () represents the total number of occurrences of the abstract state at time t in the history; Represents the current abstract state f(s) in memory M. t The most similar k states; K is the kernel function used to calculate the similarity between the two, c is a constant that guarantees the minimum spurious count; ∈ is a constant, and d represents the Euclidean distance between two vectors x and y. This represents the average value. Dividing by the average value prevents excessive differences between extracted features, thus making it more robust.

[0048] S42, the observed image s at time t-1 t-1 and target image s target The input is fed into an RND random network distillation process. The predicted and target networks output corresponding predicted and true features, respectively. The mean squared error of both is then calculated to obtain the long-term intrinsic reward throughout the training period.

[0049] S43, based on short-term rewards and long-term intrinsic rewards Calculate the total intrinsic reward Then, the external reward obtained in step S3 The total reward r is obtained by adding them together. t :

[0050]

[0051]

[0052] S44, based on the total reward r t Calculate A3C loss L a3c And the prediction loss L of the depth map and semantic segmentation map obtained in step S22. deconv The sum of these values ​​yields the total loss value L. This total loss is then used for backpropagation to update the network parameters of the feature extraction module and the deep reinforcement learning model. The detailed calculation formula is as follows:

[0053] R = r t +γr t+1 +γ 2 r t+2 ...+γT-t r T

[0054]

[0055] L = L a3c +L deconv

[0056] Where, r t r t+1 r t+2 Represents the reward value at the corresponding moment, r T Let γ be the value of the state at time T (representing reaching the destination or reaching the set number of steps); γ is the discount factor in reinforcement learning, representing a discount of the future reward to the current time; R is the discounted reward of the state at time t, representing the true value of the state at time t; L is the value of the state at time t. a3c V(s) represents the loss value of the deep reinforcement learning model, where α is the weight of the value loss, β is the weight of the entropy loss, and V(s) is the value of the deep reinforcement learning model. t ) represents the value network at time t, specifically the state s. t The predictive value of π(a) t |s t ) indicates that in state s t The actions taken, Entropy loss The entropy loss of the policy network is represented by the pytoch entropy function, which is calculated using pytoch.

[0057] Navigation is performed in various indoor environments. The reward sparsity problem is solved by using intrinsic reward signals, which accelerates the convergence speed of the model, trains the model, and improves navigation performance.

[0058] Furthermore, in step S42, the long-term intrinsic reward throughout the training period is calculated. The method is as follows:

[0059] Observation image s t-1 and target image s target Convert to grayscale image, standardize the grayscale image corresponding to the observation image, and then crop the pixel values ​​in the image;

[0060] The processed observation image and target image are concatenated using a cat operation and then input into the target network and prediction network of the RND, respectively, to obtain the target feature h(s). t ,s target ) and predictive features The weights of the target network are randomly initialized and fixed, and this part of the network is not trained. The prediction network is responsible for predicting the output value of the target network, and its weight parameters need to be updated. This involves distilling the randomized weights of the target network into the prediction network, calculating the mean squared error (MSELoss) using the predicted features and the true features, and then performing interval processing on this error to obtain the long-term intrinsic reward.

[0061]

[0062] Where N represents the maximum reward scale, since long-term reward represents a multiple of short-term reward, so the range of the result is controlled to [1, N] using min.

[0063] Divide intrinsic rewards into long-term and short-term rewards to learn a better exploration strategy.

[0064] The present invention also provides a visual navigation system based on goal-driven and intrinsic motivation exploration, including a processor and a memory. The memory is used to store a deep reinforcement learning model and a program based on the method of the present invention. The processor calls the program in the memory to perform visual navigation.

[0065] This system does not require any pre-trained models or rely on any external knowledge bases. It takes visual observation images and target images as input and directly predicts the optimal action for each step to complete the visual navigation task. Attached Figure Description

[0066] Figure 1 This is a flowchart illustrating the visual navigation method based on goal-driven and intrinsic motivation exploration according to the present invention.

[0067] Figure 2 This is a framework diagram of the visual navigation method based on goal-driven and intrinsic motivation exploration of the present invention;

[0068] Figure 3 This is a schematic diagram of the deep reinforcement learning model of the visual navigation method based on goal-driven and intrinsic motivation exploration of the present invention;

[0069] Figure 4 This is a schematic diagram of the RND network structure of the visual navigation method based on goal-driven and intrinsic motivation exploration of the present invention. Detailed Implementation

[0070] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0071] In the description of this invention, it should be understood that the terms "longitudinal", "lateral", "up", "down", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0072] In the description of this invention, unless otherwise specified and limited, it should be noted that the terms "installation", "connection" and "linking" should be interpreted broadly. For example, they can refer to mechanical or electrical connections, or internal connections between two components. They can be direct connections or indirect connections through an intermediate medium. Those skilled in the art can understand the specific meaning of the above terms according to the specific circumstances.

[0073] This invention discloses a visual navigation method based on goal-driven and intrinsic motivation exploration. It uses only visually observed images and target images as input to directly predict the optimal action for each step to complete the visual navigation task. For example... Figure 1 and Figure 2 As shown, the visual navigation method includes the following steps:

[0074] S1 simulates a mobile robot traversing every state in the AI2-THOR simulation environment, generating an offline dataset for the AI2-THOR simulation platform.

[0075] S2, input the observed image and target image at the current moment into the feature extraction module to extract the feature vector at the current moment;

[0076] S3. Input the feature vector obtained in step S2 into a deep reinforcement learning model (such as A3C network model, PPO model, etc.) to predict the optimal navigation action and interact with the environment to obtain the observation image and external reward at the next moment.

[0077] S4, input the observed image and the feature vector from step S2 into the intrinsic reward calculation module to obtain the intrinsic reward, and calculate the loss value by combining it with the extrinsic reward. Use the loss value to train a deep reinforcement learning model (such as an A3C network model), as follows. Figure 3 As shown.

[0078] In a preferred embodiment of the present invention, the method for generating the offline dataset of the AI2-THOR simulation platform in step S1 is as follows:

[0079] S11, Download the Python file for constructing the offline environment, set the cell size after meshing in the AI2-THOR simulation environment, the image size observed at each coordinate, and take multiple scenes of different spatial types. Use all scenes as the training set and take one scene from each space as the test set. For example, take 5 scenes from 4 different room types (bathroom, living room, kitchen, bedroom) for a total of 20 scenes as the training set, and take 1 scene from each of them for a total of 4 scenes as the test set.

[0080] S12 simulates a mobile robot traversing each coordinate in the AI2-THOR simulation environment, simultaneously acquiring the corresponding RGB image, RGB-D depth image, and RGB-S semantic segmentation image. A JSON file stores the transformation relationships between coordinates, and an HDF5 file stores the corresponding image information, generating an offline dataset. The meshed cell size of the AI2-THOR simulation environment is 0.5, the camera height is 0.675, and the size of the image observed at each coordinate is 172×172×3; the RGB image size is 172×172×3, the RGB-D depth image size is 172×172×1, and the RGB-S semantic segmentation image size is 172×172×3.

[0081] In a preferred embodiment of the present invention, the method for extracting the feature vector at the current time in step S2 is as follows:

[0082] S21, view the current observation image s t (size 172×172×3) and target image s target (Size 172×172×3), respectively input into the Siamese convolutional neural network to obtain the feature vectors of the two, and then use convolutional network operations to fuse the two feature vectors to obtain the fused feature vector of the observed image and the target image;

[0083] S22, In order to enable the convolutional network in the previous step to extract depth and semantic information, this invention replaces the direct input of depth image and semantic segmentation image, and instead inputs the fused features of observation and target image into the designed deconvolutional neural network to predict the depth map, semantic segmentation map of observation image and semantic segmentation map of target image, and calculates the loss with the real values ​​in offline dataset.

[0084] The fused feature vectors are input into three deconvolutional neural networks to predict the depth map of the observed image. Semantic segmentation graph semantic segmentation map of the target image Then, the PyTorch built-in 2D average pooling function `avg_pool2d` is used to perform average pooling on the depth map and semantic segmentation map in the offline dataset to obtain the ground truth value `d`, which is the same size as the predicted map. s :42×42×1、s s :42×42×3、s target :42×42×3, then use the mean squared error MSELoss to calculate the total loss value L of the prediction result. deconv The specific formula is as follows;

[0085]

[0086] S23. The fused features (size 32×9×9) obtained in step S21 are input into a fully connected layer with 512 hidden units and ReLU activation function. Then, the features are concatenated with the action distribution features (size 4×1) and normalized reward features (size 1×1) from the previous time step using the cat feature concatenation method. The features are then input into a Long Short-Term Memory (LSTM) network with 512 hidden units and 1 layer to extract a feature vector (size 512×1) containing historical features.

[0087] Among them, the action distribution feature indicates which action was selected in the previous moment. Taking the four actions of forward, backward, left turn, and right turn as an example, if the action of "forward" was selected in the previous moment, the action distribution feature is a 4×1 vector [1,0,0,0]. The reward feature is a 1×1 vector between [0-1] after normalizing the external reward obtained in the previous moment. The hidden state feature is an output of the long short-term memory network in the previous moment, representing the historical state feature. In this embodiment, the size of this vector is 512×1.

[0088] Preferably, the Siamese convolutional neural network includes a first two-dimensional convolutional layer containing 32 filters, a 7×7 convolutional kernel, a span of 4, and an activation function of ReLU, and a second two-dimensional convolutional layer containing 32 filters, a 4×4 convolutional kernel, a span of 2, and an activation function of ReLU.

[0089] The current observation image s t and target image s target The first two-dimensional convolutional layer and the second two-dimensional convolutional layer are input sequentially to obtain the observed image s. t and target image s target The feature vector (size 32×20×20);

[0090] Use feature stitching operations (such as cat, stack, etc.) to stitch the observed image s t and target image s targetThe feature vectors are concatenated to obtain a concatenated vector (size 64×20×20). Then, a two-dimensional convolutional layer containing 64 filters, a 4×4 convolutional kernel, a span of 2, and a ReLU activation function is sequentially input, along with a two-dimensional convolutional layer containing 32 filters, a 1×1 convolutional kernel, a span of 1, and a ReLU activation function, to obtain a fused feature vector of the observed and target images (size 32×9×9).

[0091] More preferably, the deconvolutional neural network includes a third two-dimensional deconvolutional layer with 16 filters, a 4×4 convolutional kernel, a span of 2, and a ReLU activation function; a fourth two-dimensional deconvolutional layer with 1 filter, a 4×4 convolutional kernel, and a span of 2; a fifth two-dimensional deconvolutional layer with 16 filters, a 4×4 convolutional kernel, a span of 2, and a ReLU activation function; and a sixth two-dimensional deconvolutional layer with 3 filters, a 4×4 convolutional kernel, and a span of 2.

[0092] The fused feature vectors are passed through the third and fourth two-dimensional deconvolution layers to predict the depth map of the observed image (42×42×1 in size).

[0093] The fused feature vectors are passed through the fifth and sixth two-dimensional deconvolution layers to predict semantic segmentation maps (42×42×3 in size) of the observed and target images.

[0094] In a preferred embodiment of the present invention, the method for obtaining the observation image and external reward at the next time step in step S3 is as follows:

[0095] S31, input the fused feature vector obtained in step S23 into the A3C network to predict the agent's optimal navigation action at that moment;

[0096] S32, the agent executes the optimal navigation action, using the offline dataset to move to the coordinates of the next time step and obtain the corresponding observation image s. t+1 Then, based on the reward function, calculate the external reward at that moment.

[0097]

[0098] More preferably, the A3C network includes a policy network with 4 hidden units and a value network with 1 hidden unit (the number of selectable navigation actions corresponds to the number of hidden units in the policy network; in this embodiment, 4 navigation actions are used).

[0099] The fused feature vectors are input into the policy network and the value network respectively. The value network outputs the value at that moment. A softmax operation is performed on the output of the policy network to obtain the probability of each action being selected. The action with the highest probability is selected as the optimal navigation action at that moment.

[0100] In a preferred embodiment of the present invention, in the intrinsic reward calculation module of step S4, to prevent some states that have lost novelty during the exploration process (which may lead to the optimal path) from being visited again, the intrinsic reward is divided into long-term (the entire training period) and short-term (within one iteration) intrinsic rewards. This prevents the agent from visiting the same state in the short term and gradually prevents it from visiting the same state in the long term, ensuring that it does not get trapped in local optima and improving navigation performance. Figure 2 and Figure 4 As shown, the specific steps include the following:

[0101] S41, input the fused feature vector obtained in step S2 into the short-term reward calculation module to obtain the short-term intrinsic reward within one iteration. The short-term reward calculation module adopts a counting method based on state abstraction, using a memory M (implemented as a list in the program) to store the historical abstract states in one iteration. At time t, the stored information in M ​​is {f(s1), f(s2)...f(s... t-1 )}, where f represents the feature extraction function, s t-1 For observation images;

[0102] By compressing the original state space, a pseudo-count is performed on the abstract state of the currently observed image, and the reciprocal of the pseudo-count is used as the short-term intrinsic reward.

[0103]

[0104]

[0105] Wherein, nf(s) t () represents the total number of occurrences of the abstract state at time t in the history; Represents the current abstract state f(s) in memory M. t The most similar k states (e.g., k = 10); K is the kernel function used to calculate the similarity between the two; c is a constant that guarantees the minimum spurious count (e.g., c = 0.001); ∈ is a constant; d function represents the Euclidean distance between two vectors x and y. This represents the average value. Dividing by the average value prevents excessive differences between extracted features, thus making it more robust.

[0106] S42, the observed image s at time t-1 t-1 and target image s target Input to RND random network distillation (e.g.) Figure 4As shown (i.e., long-term reward), the long-term intrinsic reward during the entire training period is obtained by calculating the mean squared error of the predicted and true features output by the prediction and target networks, and then calculating the corresponding predicted and true features.

[0107] S43, based on short-term rewards and long-term intrinsic rewards Calculate the total intrinsic reward Then, the external reward obtained in step S3 The total reward r is obtained by adding them together. t :

[0108]

[0109]

[0110] This hybrid reward system can quickly prevent agents from accessing the same state in the short term, slowly prevent them from accessing the same state in the long term, encourage agents to access all possible states in the environment, and ensure that they do not get trapped in local optima.

[0111] S44, based on the total reward r t Calculate A3C loss L a3c And the prediction loss L of the depth map and semantic segmentation map obtained in step S22. deconv The sum of these values ​​yields the total loss value L. This total loss is then used for backpropagation to update the network parameters of the feature extraction module and the deep reinforcement learning model. The detailed calculation formula is as follows:

[0112] R = r t +γr t+1 +γ 2 r t+2 ...+γ T-t r T

[0113]

[0114] L = L a3c +L deconv

[0115] Where, r t r t+1 r t+2 Represents the reward value at the corresponding moment, r T Let γ be the value of state at time T (representing reaching the destination or reaching the set number of steps, which is 20 in this embodiment); γ is the discount factor in reinforcement learning, representing a discount of the future reward to the current time (0.99 in this embodiment); R is the discounted reward of state at time t, representing the true value of state at time t; L a3cV(s) represents the loss value of the deep reinforcement learning model, where α is the weight of the value loss (0.5 in this example), β is the weight of the entropy loss (0.01 in this example), and V(s) is the weight of the value loss. t ) represents the value network at time t, specifically the state s. t The predictive value of π(a) t |s t ) indicates that in state s t The actions taken, Entropy loss The entropy loss of the policy network is represented by the pytoch entropy function, which is calculated using pytoch.

[0116] More preferably, in step S42, the long-term intrinsic reward is calculated throughout the training period. The method is as follows:

[0117] Observation image s t-1 and target image s target Convert to grayscale image (size 172×172×1), standardize the grayscale image corresponding to the observation image, and then crop the pixel values ​​in the image to (-5,5);

[0118] To eliminate the influence of different colors in the scene, the processed observation image and target image are concatenated using a cat operation, and then input into the target network and prediction network of the RND (the two networks have identical structures) respectively, to obtain the target feature h(s). t ,s target ) and predictive features The weights of the target network are randomly initialized and fixed, and this part of the network is not trained. The prediction network is responsible for predicting the output value of the target network, and its weight parameters need to be updated. This involves distilling the randomized weights of the target network into the prediction network, calculating the mean squared error (MSELoss) using the predicted features and the true features, and then performing interval processing on this error to obtain the long-term intrinsic reward.

[0119]

[0120] Where N represents the maximum reward scale, since long-term reward represents a multiple of short-term reward, so the range of the result is controlled to [1, N] using min.

[0121] Divide intrinsic rewards into long-term and short-term rewards to learn a better exploration strategy.

[0122] The present invention also provides a visual navigation system based on goal-driven and intrinsic motivation exploration, including a processor and a memory. The memory is used to store a deep reinforcement learning model and a program based on the method described in the present invention. The processor calls the program in the memory to perform visual navigation.

[0123] This technical solution designs a fully end-to-end visual navigation framework that takes only the observed image and the target image as input and directly outputs the predicted optimal navigation action. It includes auxiliary tasks that predict the depth map and semantic segmentation map of the observed image and the target image respectively, enabling the deep reinforcement learning model to extract useful depth and semantic features. This improves navigation capabilities in simulated indoor environments, solves the reward sparsity problem in navigation, and increases the model's convergence speed, allowing it to achieve a higher navigation success rate in a shorter time.

[0124] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0125] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

Claims

1. A visual navigation method based on goal-driven and intrinsic motivation exploration, characterized in that, Includes the following steps: S1 simulates a mobile robot traversing every state in the AI2-THOR simulation environment, generating an offline dataset for the AI2-THOR simulation platform. S2, input the observed image and target image at the current moment into the feature extraction module to extract the feature vector at the current moment; S3. Input the feature vector obtained in step S2 into the deep reinforcement learning model to predict the optimal navigation action and interact with the environment to obtain the observation image and external reward at the next moment. S4, input the observed image, the target image and the feature vector from step S2 into the intrinsic reward calculation module to obtain the intrinsic reward, and calculate the loss value by combining it with the external reward. Use the loss value to update the network parameters of the feature extraction module and the deep reinforcement learning model. The method for extracting the feature vector at the current time step S2 is as follows: S21, display the observed image at the current moment. and target image The two feature vectors are input into the Siamese convolutional neural network respectively to obtain their feature vectors. The two feature vectors are then fused using convolutional network operations to obtain the fused feature vector of the observed image and the target image. S22, the fused feature vector is input into three deconvolutional neural networks to predict the depth map of the observed image. Semantic segmentation graph semantic segmentation map of the target image Then, the depth map and semantic segmentation map in the offline dataset are averaged using PyTorch's built-in 2D average pooling function avg_pool2d to obtain ground truth values ​​of the same size as the prediction map. , , Then, the total loss value of the prediction result is calculated using the mean squared error (MSELoss). The specific formula is as follows; ； S23, the fusion features obtained in step S21 are concatenated with the action distribution features and normalized reward features of the previous time step, and then input into the long short-term memory network along with the hidden state features of the previous time step to extract feature vectors containing historical features. In the intrinsic reward calculation module of step S4, the intrinsic reward is divided into long-term and short-term intrinsic rewards. This prevents the agent from accessing the same state in the short term and gradually prevents access to the same state in the long term. Specifically, this includes the following steps: S41, input the fused feature vector obtained in step S2 into the short-term reward calculation module to obtain the short-term intrinsic reward within one iteration. The short-term reward calculation module adopts a counting method based on state abstraction, using a memory M to store the historical abstract state in one iteration. At time t, the information stored in M ​​is... , Represents the feature extraction function. For observation images; By compressing the original state space, a pseudo-count is performed on the abstract state of the currently observed image, and the reciprocal of the pseudo-count is used as the short-term intrinsic reward. : ， ， in, This represents the total number of occurrences of the abstract state at time t in the history; Represents the memory M and the current abstract state. The k most similar states; K is the kernel function used to calculate the similarity between the two, and c is a constant that guarantees the minimum spurious count; It is a constant, and the d function represents the Euclidean distance between two vectors x and y. This represents the average value. Dividing by the average value prevents excessive differences between extracted features, thus making it more robust. S42, the observed image at time t-1 and target image The input is fed into an RND random network distillation process. The predicted and target networks output corresponding predicted and true features, respectively. The mean squared error of both is then calculated to obtain the long-term intrinsic reward throughout the training period. ; S43, based on short-term rewards and long-term intrinsic rewards Calculate the total intrinsic reward Then, combined with the external reward obtained in step S3 Add them together to get the total reward : ， ； S44, based on total reward Calculate A3C loss And the prediction loss of the depth map and semantic segmentation map obtained in step S22. The sum is the total loss value. The network parameters of the feature extraction module and the deep reinforcement learning model are updated by backpropagation through the total loss: ， ， ， in, This represents the reward value at the corresponding moment; For terminal time The value of a state indicates whether the destination has been reached or the set number of steps has been taken. To reinforce the discount factor in learning, it represents a discount of the future reward to the current moment; Let $t$ be the discounted reward for the state at time $t$, representing the true value of the state at time $t$. The loss value for the deep reinforcement learning model. As a weight for value loss, The weights for entropy loss, Represents the state of the value network at time t. The predictive value Indicates the state The actions taken The entropy loss of the policy network is represented by the pytoch entropy function, which is calculated using pytoch. In step S42, the long-term intrinsic reward is calculated throughout the training period. The method is as follows: Observation image and target image Convert to grayscale image, standardize the grayscale image corresponding to the observation image, and then crop the pixel values ​​in the image; The processed observation image and target image are concatenated using a cat operation and then input into the target network and prediction network of the RND, respectively, to obtain the target features. and predictive features The target network's weights are randomly initialized and fixed, and this part of the network is not trained. The prediction network is responsible for predicting the output value of the target network, requiring weight parameter updates. This involves distilling the randomized target network weights into the prediction network, calculating the mean squared error (MSELoss) using predicted and true features, and then performing interval processing on this error to obtain the long-term intrinsic reward. : ， Where N represents the maximum reward scale, since long-term reward represents a multiple of short-term reward, so the range of the result is controlled to [1, N] using min.

2. The visual navigation method based on goal-driven and intrinsic motivation exploration as described in claim 1, characterized in that, The method for generating the offline dataset for the AI2-THOR simulation platform in step S1 is as follows: S11, Download the Python file to construct the offline environment, set the cell size after meshing of the AI2-THOR simulation environment and the image size parameters observed at each coordinate, take multiple scenes of different spatial types, use all scenes as the training set, and take one scene from each space as the test set; S12 simulates a mobile robot traversing each coordinate in the AI2-THOR simulation environment, simultaneously acquiring the corresponding RGB image, RGB-D depth image, and RGB-S semantic segmentation image, saving the transformation relationship between each coordinate, saving the image information corresponding to the coordinate, and generating an offline dataset.

3. The visual navigation method based on goal-driven and intrinsic motivation exploration as described in claim 1, characterized in that, The twin convolutional neural network includes a first two-dimensional convolutional layer with 32 filters, a 7×7 convolutional kernel, a span of 4, and a ReLU activation function, and a second two-dimensional convolutional layer with 32 filters, a 4×4 convolutional kernel, a span of 2, and a ReLU activation function. The observed image at the current moment and target image The observed image is obtained by sequentially inputting the first two-dimensional convolutional layer and the second two-dimensional convolutional layer. and target image eigenvectors; Use feature stitching operations on the observed image and target image The feature vectors are concatenated and then sequentially input into a two-dimensional convolutional layer containing 64 filters, a 4×4 convolutional kernel, a span of 2, and a ReLU activation function, and a two-dimensional convolutional layer containing 32 filters, a 1×1 convolutional kernel, a span of 1, and a ReLU activation function, to obtain the fused feature vector of the observed image and the target image.

4. The visual navigation method based on goal-driven and intrinsic motivation exploration as described in claim 1, characterized in that, The three deconvolutional neural networks include a third two-dimensional deconvolutional layer with 16 filters, a 4×4 convolutional kernel, a span of 2, and a ReLU activation function; a fourth two-dimensional deconvolutional layer with 1 filter, a 4×4 convolutional kernel, and a span of 2; a fifth two-dimensional deconvolutional layer with 16 filters, a 4×4 convolutional kernel, a span of 2, and a ReLU activation function; and a sixth two-dimensional deconvolutional layer with 3 filters, a 4×4 convolutional kernel, and a span of 2. The fused feature vectors are passed through the third and fourth two-dimensional deconvolution layers to predict the depth map of the observed image. The fused feature vectors are passed through the fifth and sixth two-dimensional deconvolution layers to predict semantic segmentation maps of the observed and target images.

5. The visual navigation method based on goal-driven and intrinsic motivation exploration as described in claim 1, characterized in that, The method for obtaining the observation image and external reward at the next time step in step S3 is as follows: S31, Input the fused feature vector obtained in step S2 into the A3C network to predict the agent's optimal navigation action at that moment; S32, the agent executes the optimal navigation action, using the offline dataset to move to the coordinates of the next time step and obtain the corresponding observation image. Then, based on the reward function, calculate the external reward at that moment. : 。 6. The visual navigation method based on goal-driven and intrinsic motivation exploration as described in claim 5, characterized in that, The A3C network includes a policy network with four hidden units and a value network with one hidden unit. The fused feature vectors are input into the policy network and the value network respectively. The value network outputs the value at that moment. A softmax operation is performed on the output of the policy network to obtain the probability of each action being selected. The action with the highest probability is selected as the optimal navigation action at that moment.

7. A visual navigation system based on goal-driven and intrinsic motivation exploration, characterized in that, It includes a processor and a memory, the memory being used to store a deep reinforcement learning model and a program based on the method of any one of claims 1-6, the processor calling the program in the memory to perform visual navigation.

Images