Indoor robot object navigation method based on spatio-temporal context sensing converter

Abstract

The invention discloses an indoor robot object navigation method based on a spatio-temporal context sensing converter. A navigation encoder and a navigation decoder are included. The navigation encoder comprises three multi-layer convolutional neural networks, not only can encode the scene space layout and the time sequence pose track of the robot under the view angle of the aerial view, but also can encode a new feasible boundary diagram to enhance the understanding of the navigation model for potential high-value explorable region clues. The navigation decoder comprises a feature enhancement module based on a self-attention mechanism, and a space boundary encoder and a time sequence pose encoder based on a cross attention mechanism. The two sub-decoder modules construct a long-distance spatio-temporal context state representation between the spatial layout and the time sequence pose by using three encoding features output by the navigation encoder. According to the method, a relatively high success rate is achieved in different indoor scene data sets.

Description

Technical Field

[0001] The present invention belongs to the field of embodied intelligence, and in particular, is an indoor robot object navigation method based on a spatiotemporal context-aware converter. Background Art

[0002] As a key branch of embodied artificial intelligence, object navigation provides an important foundation for various vision-based navigation-related applications, such as image-based target navigation and vision-language navigation. In object navigation, a robot learns to recognize an unknown 3D environment from RGB-D images from a first-person perspective and enables its built-in navigation model to make intelligent decisions to locate a specified target. Therefore, how to fully exploit the spatiotemporal navigation cues obtained from the explored spatial layout and temporal agent pose trajectories and use them to accurately guide navigation decisions has become the key to the success of object navigation tasks.

[0003] Existing object navigation methods can be mainly divided into two categories: end-to-end methods and modular methods. End-to-end methods directly process RGB images from the first-person perspective and find the target object by learning the mapping relationship between the current state and the optimal short-term action. Although this method is effective, when there is a significant difference between the unknown scene layout and the prior knowledge, the learned navigation model will inevitably fall into suboptimal decisions, resulting in frequent collisions with surrounding objects or obstacles. In contrast, modular methods use the scene information accumulated in the map under the bird's-eye view to determine the long-term goal and search for the target object. For example, a simple network like a convolutional neural network is used to fuse the accumulated navigation clues, including the explored spatial layout and temporal pose trajectory, to form the current robot state representation. However, due to the large spatial layout in different scenes and the inherent local bias of convolutional neural networks, the simple navigation information fusion based on convolutional neural networks may cause the navigation model to ignore the long-term spatiotemporal dependency between the spatial layout and the temporal pose trajectory, resulting in misinterpretation of the state representation and potential decision failure. Summary of the invention

[0004] The purpose of the present invention is to propose an indoor robot object navigation method based on a spatiotemporal context-aware converter to fully aggregate discriminative spatiotemporal context information to achieve reliable object navigation.

[0005] The technical solution to achieve the purpose of the present invention is: an indoor robot object navigation method based on a spatiotemporal context perception converter, comprising the following steps:

[0006] Step 1: The robot obtains the color image and depth map from the current first-person perspective from the scene. The semantic segmentation network segments different object instances from the color image, and all pixels in the depth map are converted into a three-dimensional point cloud.

[0007] In step 2, the pixel categories of each object obtained by the semantic segmentation network in step 1 are aligned with the points at the corresponding positions in the point cloud, and the point cloud with semantic labels is projected to the bird's-eye view using geometric transformation to obtain the spatial layout and the temporal trajectory of the robot.

[0008] Step 3: Use the image processing tool OpenCV to extract the outline of the occupancy map in the spatial layout, dilate the outer boundary of the obstacle map in the spatial layout, subtract the processed obstacle map from the occupancy map, and then use a scoring function to screen out important feasible boundaries to construct a feasible boundary map.

[0009] Step 4: Input the feasible boundary map obtained in step 3, the spatial layout obtained in step 2, and the temporal trajectory of the robot into the navigation encoder to encode boundary features, spatial layout features, and temporal trajectory features respectively;

[0010] In step 5, the three features obtained in step 4 are input into the navigation decoder to construct a more accurate long-distance spatiotemporal context state representation, and this state representation is used to infer a reasonable long-term goal to guide the robot to explore unknown scenes until the target object is found.

[0011] Furthermore, the semantic segmentation network used in step 1 is a general residual encoder-decoder network.

[0012] Furthermore, the candidate feasible boundary map is obtained by subtracting the obstacle map and the occupancy map in step 3. H and W represent the boundary graph M b Width and height, M b contains all candidate boundaries, using the scoring function S FO (f i ) b All boundaries are preliminarily screened in the scoring function S FO (f i ) works as follows:

[0013] S FO (f i )=A fi +λ·dist(p t ,c fi )#(1)

[0014]

[0015] Among them, λ represents a constant, p t Indicates the current position coordinates of the robot, f i Indicates M b One of the boundaries, Represents the boundary fi The center position coordinates of Represents the boundary f i The number of pixels occupied; merge the boundaries with scores higher than the set threshold Th into a feasible boundary map

[0016] Furthermore, the spatial layout in step 4 includes the obstacle map Occupancy map and an object semantic graph containing k kinds of objects The robot's time-series trajectory contains the robot's motion trajectory diagram and the robot's current position map The feasible boundary graph M is encoded using the boundary encoder E1 in the navigation encoder f And get the boundary features B s Represents feature b t The number of channels, H2 and W2 represent feature b t The width and height of the spatial layout encoder E2 encodes the obstacle map M o , occupancy map M s and object semantic graph M c The spatial layout of K s Represents feature m t The number of channels; the temporal pose encoder E3 encodes the robot's motion trajectory M a and the robot's current position map M l Combined with target semantic features Merge to get the time series trajectory features A s Represents feature m t The number of channels.

[0017] Furthermore, in step 5, the navigation decoder is used to decode the boundary feature b t 、Spatial layout characteristics m t and the time series trajectory features e t Decode; first use the self-attention mechanism to decode the spatial layout feature m t and the time series trajectory features e t Perform feature enhancement processing:

[0018]

[0019] in are all learnable parameters. and They are feature m t and feature e t After the enhancement, the size of the features is the same as mt and e t In the navigation decoder, the spatial boundary decoder uses b t and Calculate the important boundary weights hd3 represents the number of heads of the self-attention mechanism in formula (5):

[0020]

[0021] The temporal pose decoder uses and Calculate the target-related weight hd4 represents the number of heads of the self-attention mechanism in formula (6):

[0022]

[0023] Important boundary weight W f , target-related weight W s and time series trajectory characteristics Together they are used to construct a temporal context state representation:

[0024]

[0025] s t After processing through a multi-layer perceptron network MLP, an action distribution about the long-term goal is obtained Finally, according to a t The probability distribution of the scene is used to randomly sample a vertex from the eight vertices of the bird's-eye view (including the four corner points of the bird's-eye view and the midpoints of the four boundaries of the bird's-eye view) as the long-term goal that the robot needs to explore in the next step. During the process of the robot navigating to the long-term goal, if the semantic segmentation network in step 2 detects the target object, the robot will use the target object as the target point for the next navigation, navigate to the vicinity of the target object, and end the navigation process.

[0026] Compared with the prior art, the present invention has the following significant advantages:

[0027] (1) We construct a novel navigation transformer to efficiently aggregate spatiotemporal contextual navigation cues to support discriminative robot state representation and reliable decision making.

[0028] (2) The present invention develops a navigation encoder enhanced with a traversable boundary map, which is used to encode the spatial layout, temporal pose, and spatiotemporal features of the traversable boundary map. In particular, the present invention innovatively constructs and integrates an efficient traversable boundary map to identify the most valuable areas in the explored scene.

[0029] (3) We design a navigation decoder that leverages spatial layout, temporal pose trajectories, and traversable boundaries to build a better spatiotemporal contextual state representation. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is the overall framework diagram of the method of the present invention.

[0031] Figure 2 It is a schematic diagram for constructing a traversable boundary graph.

[0032] Figure 3 It is a comparison chart between the method of the present invention and the object navigation method based on a simple convolutional neural network. DETAILED DESCRIPTION

[0033] The present invention provides an indoor robot object navigation method based on a spatiotemporal context perception converter, and the specific steps are as follows:

[0034] Step 1: The robot obtains the RGB color image and depth map from the current first-person perspective from the scene. The residual encoder-decoder network segments different object instances from the color image and converts all pixels in the depth map into a 3D point cloud;

[0035] Step 2: Align each object pixel category obtained by the semantic segmentation network in step 1 with the point at the corresponding position in the point cloud, and use geometric transformation to project the point cloud with semantic labels to the bird's-eye view to obtain the explored spatial layout and the robot's temporal trajectory.

[0036] Step 3: Use the image processing tool OpenCV to extract the outline of the occupancy map in the spatial layout, dilate the outer boundary of the obstacle map in the spatial layout, and subtract the processed obstacle map from the occupancy map to obtain the candidate feasible boundary map. H and W represent the boundary graph M b Width and height. M b All candidate boundaries are included, and then the scoring function S is used FO (f i ) b All boundaries are preliminarily screened in the scoring function S FO (f i ) works as follows:

[0037]

[0038]

[0039] Where λ is a constant, Indicates the current position coordinates of the robot, f i Indicates M bOne of the boundaries, Represents the boundary f i The center position coordinates of Represents the boundary f i The number of pixels occupied. Merge the boundaries with higher scores into a feasible boundary map

[0040] Step 4: The spatial layout obtained in step 2 includes the obstacle map Occupancy map and an object semantic graph containing k kinds of objects The robot's time-series trajectory contains the robot's motion trajectory diagram and the robot's current position map The feasible boundary graph M is encoded using the boundary encoder E1 in the navigation encoder f And get the boundary features B s It represents feature b t The number of channels; the spatial layout encoder E2 encodes the obstacle map M o , occupancy map M s and object semantic graph M c The spatial layout of K s It represents the feature m t The number of channels; the temporal pose encoder E3 encodes the robot's motion trajectory M a and the robot's current position map M l Combined with target semantic features Merge to get the time series trajectory features A s It represents the characteristic e t The number of channels.

[0041] Step 5: Use the navigation decoder to decode the boundary feature b t , spatial layout feature m t and the time series trajectory features e t First, the self-attention mechanism is used to decode the spatial layout feature m t and the time series trajectory features e t Perform feature enhancement processing:

[0042]

[0043] in are all learnable parameters. In the navigation decoder, the spatial boundary decoder uses b t and Calculate the important boundary weights

[0044]

[0045] The temporal pose decoder uses and Calculate the target-related weight

[0046]

[0047] Important boundary weight W f , target related weight W s and time series trajectory characteristics Together they are used to construct a temporal context state representation:

[0048]

[0049] s t After processing through a multi-layer perceptron network (MLP), an action distribution about the long-term goal is obtained Finally, according to a t The probability distribution of the scene is used to randomly sample a vertex from the eight vertices of the bird's-eye view (including the four corner points of the bird's-eye view and the midpoints of the four boundaries of the bird's-eye view) as the long-term goal that the robot needs to explore in the next step. During the process of the robot navigating to the long-term goal, if the semantic segmentation network in step 2 detects the target object, the robot will use the target object as the target point for the next navigation, navigate to the vicinity of the target object, and end the navigation process.

[0050] The navigation converter in the present invention mainly consists of a navigation encoder enhanced by a traversable boundary map and a spatiotemporal navigation decoder. In the navigation encoder, in addition to encoding the spatial layout and temporal position trajectory, the present invention also generates an effective traversable boundary map to identify the area where the exploration is most likely to find the target. Therefore, the present invention encodes this boundary map to enhance the original feature representation with rich exploration guidance. With the above-mentioned encoded spatial and temporal features, the goal of the spatiotemporal navigation decoder in the present invention is to learn the spatiotemporal contextual associations between the spatial layout, the temporal posture trajectory and the traversable frontier to achieve reliable navigation information fusion and discriminative state representation. Specifically, the self-attention module is used to strengthen the spatial and temporal attributes of the spatial layout and the temporal posture trajectory. In addition, based on the cross-attention mechanism, the spatial boundary decoder uses the current spatial layout features and boundary features to obtain important boundary weights. The temporal posture decoder uses the temporal posture trajectory features and the current spatial layout features to derive the target-related weights. Finally, the spatiotemporal context state representation combined with the important boundary weights and the target-related weights is considered to guide the navigation model to make reasonable decisions.

[0051] The present invention is further described below in conjunction with the accompanying drawings and embodiments of the specification. The described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0052] The present invention is described in detail below with reference to the accompanying drawings. The overall framework of the method is as follows: Figure 1 As shown, an indoor robot object navigation method based on a spatiotemporal context perception converter, the specific execution steps are:

[0053] Step 1: Figure 1 The RGB color image and depth map on the left are the scene information currently obtained by the robot. The residual encoder-decoder network segments different object instances from the color image and converts all pixels in the depth map into a 3D point cloud;

[0054] Step 2: Figure 1 The left side also shows the partial conversion process of the color pictures and depth maps collected by the robot to the spatial layout and time-series trajectory from a bird's-eye view. The semantic segmentation network aligns the pixel categories of each object with the points at the corresponding positions in the point cloud, and uses geometric transformation to project the point cloud with semantic labels to the bird's-eye view to obtain the explored spatial layout and the robot's time-series trajectory.

[0055] Step 3: Figure 2 The part of the construction process of the feasible boundary map is shown in the figure. The image processing tool OpenCV is used to extract the outline of the occupancy map in the spatial layout, and the outer boundary of the obstacle map in the spatial layout is expanded. The processed obstacle map and the occupancy map are subtracted to obtain the candidate feasible boundary map. H and W represent the boundary graph M b Width and height. M b All candidate boundaries are included, and then the scoring function S is used FO (f i ) b All boundaries are preliminarily screened in the scoring function S FO (f i ) works as follows:

[0056]

[0057] Where λ is a constant, Indicates the current position coordinates of the robot, f i Indicates M b One of the boundaries, Represents the boundary f i The center position coordinates of Represents the boundary f i The number of pixels occupied. Merge the boundaries with higher scores into a feasible boundary map

[0058] Step 4: Figure 1 The structure and working process of the navigation encoder are shown in the figure, and the spatial layout includes an obstacle map. Occupancy map and an object semantic graph containing k kinds of objects The robot's time-series trajectory contains the robot's motion trajectory diagram and the robot's current position map The feasible boundary graph M obtained in step 1 is encoded using the boundary encoder E1 in the navigation encoder f And get the boundary features B s Represents feature b t The number of channels, H2 and W2 represent feature b t The width and height of the spatial layout encoder E2 encodes the obstacle map M o , occupancy map M s and object semantic graph M c The spatial layout of K s Represents feature m t The number of channels; the temporal pose encoder E3 encodes the robot's motion trajectory M a and the robot's current position map M l Combined with target semantic features Merge to get the time series trajectory features A s Representation feature e t The number of channels.

[0059] Step 4: Figure 1 The structure and working process of the navigation decoder are shown in the figure. The navigation decoder is used to decode the boundary feature b t , spatial layout feature m t and the time series trajectory features e t First, the self-attention mechanism is used to decode the spatial layout feature m t and the time series trajectory features e t Perform feature enhancement processing:

[0060]

[0061] in are all learnable parameters. In the navigation decoder, the spatial boundary decoder uses b t and Calculate the important boundary weights hd3 represents the number of heads of the self-attention module in formula (5):

[0062]

[0063] The temporal pose decoder uses and Calculate the target-related weight hd4 represents the number of heads of the self-attention module in formula (6):

[0064]

[0065] Important boundary weight W f , target related weight W s and time series trajectory characteristics Together they are used to construct a temporal context state representation:

[0066]

[0067] s t After processing through a multi-layer perceptron network (MLP), an action distribution about the long-term goal is obtained Finally, according to a t The probability distribution of the scene is used to randomly sample a vertex from the eight vertices of the bird's-eye view (including the four corner points of the bird's-eye view and the midpoints of the four boundaries of the bird's-eye view) as the long-term goal that the robot needs to explore in the next step. During the process of the robot navigating to the long-term goal, if the semantic segmentation network in step 2 detects the target object, the robot will use the target object as the target point for the next navigation, navigate to the vicinity of the target object, and end the navigation process.

[0068] In order to further verify the feasibility and effectiveness of the method of the present invention, the technical effect of the present invention is further described in combination with experiments. The hardware platform used in the experiment of the present invention is: Intel (R) Xeon (R) Gold 6230CPU @ 2.10GHz × 32, 80G memory, 2 TITAN GPUs with 24G video memory, and the software platform used in the experiment of the present invention is: Ubuntu18.04.6LTS operating system and Python3.7.12, Pytorch1.8.1 deep learning framework. Gibson, Habitat-Matterport3D and Matterport3D are used to analyze the impact of the method on the robot's object navigation task. Here, the three indicators of success rate (SR), success rate weighted by path length (SPL), and distance to goal (DTG) are used to evaluate the performance of the object navigation model.

[0069] The method of the present invention is used to perform training in a training scene of Habitat-Matterport3D. During training, the initial position of the robot in the scene and the type of the target object are randomly given. The method of the present invention is tested in the test scene of Habitat-Matterport3D. In order to test the performance of the method of the present invention, the proposed indoor robot object navigation method based on the spatiotemporal context perception converter is compared with several existing advanced methods. The comparison methods include: the ANS method proposed by Chaplot, DS et al. in their published paper "learning to explore using active neural SLAM. In Proceedings of the International Conference on Learning Representations.", the prediction and navigation to unknown target method Peanut proposed by Zhai, AJ et al. in their published paper "Peanut: Predicting and navigating to unseen targets. In Proceedings of the IEEE / CVF International Conference on Computer Vision, 10926-10935.", and the prediction and navigation to unknown target method Peanut proposed by Zhang, J. et al. in their published paper "3D-aware object goal navigation via simultaneous exploration and identification. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition." Recognition, 6672–6682.” The 3D-aware navigation method proposed in the paper “Frontier semantic exploration for visual target navigation. In Proceedings of the IEEE International Conference on Robotics and Automation, 4099–4105.” by Yu, B. et al. The FSE navigation method proposed in the paper “Frontier semantic exploration for visual target navigation. In Proceedings of the IEEE International Conference on Robotics and Automation, 4099–4105.” by Cai, W.et al. proposed the pixel-guided navigation method PixNav in their paper "Bridging zero-shot object navigation and foundation models through pixel-guided navigation skill. IEEE International Conference on Robotics and Automation, 5228-5234." and Zhang, S. et al. proposed the SGM in their paper "Imagine Before Go: Self-Supervised Generative Map for Object Goal Navigation. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 16414–16425." The quantitative results are listed in Table 1. The above-mentioned evaluation index intersection-over-union ratio is used as the evaluation standard. The larger the values ​​of the evaluation indicators SR and SPL, the better, and the smaller the value of DTG, the better. .

[0070] Table 1 Quantitative test results using the method of the present invention and the prior art on Habitat-Matterport3D

[0071] method SR(%) SPL(%) DTG(m) ANS 27.3 9.2 5.80 Peanut 58.8 29.3 3.15 3D-Aware 52.4 24.5 4.25 FSE 53.8 24.6 3.75 PixNav 37.9 20.5 - SGM 59.8 29.4 3.47 Method of the present invention 61.3 29.6 3.40

[0072] From the results in Table 1, it can be seen that the method of the present invention has achieved the best results in all three indicators, indicating that the present invention can enable the robot to better complete the object navigation task in unfamiliar scenes. The qualitative test results compared with other methods are as follows Figure 2 As shown, compared with the ANS method, the method of the present invention can guide the robot to find the target object by constructing a more ideal long-distance spatiotemporal dependency.

[0073] The above embodiments should be regarded as illustrative and non-restrictive, but the protection scope of the present invention is not limited thereto. Any modifications, equivalent substitutions and improvements made by any technician familiar with the technical field within the technical scope disclosed by the invention and within the spirit and principles of the present invention should be covered by the protection scope of the present invention.

Claims

1. A method for indoor robot object navigation based on spatiotemporal context-aware converter, characterized in that: The following steps are involved: Step 1: The robot obtains the color image and depth map from the current first-person perspective from the scene, uses a semantic segmentation network to segment different object instances from the color image, and converts all pixels in the depth map into a three-dimensional point cloud; Step 2: Align the pixel categories of each object obtained by the semantic segmentation network in step 1 with the points at the corresponding positions in the point cloud, and use geometric transformation to project the point cloud with semantic labels to the bird's-eye view to obtain the spatial layout and the temporal trajectory of the robot; Step 3: Use the image processing tool OpenCV to extract the outline of the occupancy map in the spatial layout, dilate the outer boundary of the obstacle map in the spatial layout, subtract the processed obstacle map from the occupancy map, and then use a scoring function to screen out important feasible boundaries to construct a feasible boundary map. Step 4: Input the feasible boundary map obtained in step 3, the spatial layout obtained in step 2, and the temporal trajectory of the robot into the navigation encoder to encode boundary features, spatial layout features, and temporal trajectory features respectively; In step 5, the three features obtained in step 4 are input into the navigation decoder to construct a more accurate long-distance spatiotemporal context state representation, and this state representation is used to infer a reasonable long-term goal to guide the robot to explore unknown scenes until the target object is found.

2. The indoor robot object navigation method based on spatiotemporal context-aware converter according to claim 1 is characterized in that: The semantic segmentation network used in step 1 is a general residual encoder-decoder network.

3. The indoor robot object navigation method based on spatiotemporal context-aware converter according to claim 1, characterized in that: The candidate feasible boundary map is obtained by subtracting the obstacle map and the occupancy map in step 3. H and W represent the boundary graph M b Width and height, M b contains all candidate boundaries, using the scoring function S FO (f i ) b All boundaries are preliminarily screened in the scoring function S FO (f i ) works as follows: Among them, λ represents a constant, p t Indicates the current position coordinates of the robot, f i Indicates M b One of the boundaries, Represents the boundary f i The center position coordinates of Represents the boundary f i The number of pixels occupied; merge the boundaries with scores higher than the set threshold Th into a feasible boundary map 4. The indoor robot object navigation method based on spatiotemporal context-aware converter according to claim 1, characterized in that: The spatial layout in step 4 includes the obstacle map Occupancy map and an object semantic graph containing k kinds of objects The robot's time-series trajectory contains the robot's motion trajectory diagram and the robot's current position map The feasible boundary graph M is encoded using the boundary encoder E1 in the navigation encoder f And get the boundary features B s Represents feature b t The number of channels, H2 and W2 represent feature b t The width and height of the spatial layout encoder E2 encodes the obstacle map M o , occupancy map M s and object semantic graph M c The spatial layout of K s Represents feature m t The number of channels; the temporal pose encoder E3 encodes the robot's motion trajectory M a and the robot's current position map M l Combined with target semantic features Merge to get the time series trajectory features A s Represents feature m t The number of channels.

5. The indoor robot object navigation method based on spatiotemporal context-aware converter according to claim 1, characterized in that: In step 5, the navigation decoder is used to decode the boundary feature b t 、Spatial layout characteristics m t and the time series trajectory features e t Decode; first use the self-attention mechanism to decode the spatial layout feature m t and the time series trajectory features e t Perform feature enhancement processing: in are all learnable parameters. and They are feature m t and feature e t After the enhancement, the size of the features is the same as m t and e t In the navigation decoder, the spatial boundary decoder uses b t and Calculate the important boundary weights hd3 represents the number of heads of the self-attention mechanism in formula (5): The temporal pose decoder uses and Calculate the target-related weight hd4 represents the number of heads of the self-attention mechanism in formula (6): Important boundary weight W f , target-related weight W s and time series trajectory characteristics Together they are used to construct a temporal context state representation: s t After processing through a multi-layer perceptron network MLP, an action distribution about the long-term goal is obtained Finally, according to a t The probability distribution of the scene is randomly sampled from the eight vertices of the bird's-eye view as the long-term goal that the robot needs to explore in the next step. During the process of the robot navigating to the long-term goal, if the semantic segmentation network in step 2 detects the target object, the robot will use the target object as the target point for the next step of navigation, navigate to the vicinity of the target object, and end the navigation process.

6. The indoor robot object navigation method based on spatiotemporal context-aware converter according to claim 5, characterized in that: The eight vertices of the scene bird's-eye view include four corner points of the bird's-eye view and midpoints of four boundaries of the bird's-eye view.

Images