An indoor robot physical object navigation method based on space-time context perception converter

Abstract

This invention discloses an indoor robot object navigation method based on a spatiotemporal context-aware converter, comprising a navigation encoder and a navigation decoder. The navigation encoder includes three multi-layer convolutional neural networks, which not only encode the scene spatial layout from a bird's-eye view and the robot's temporal pose trajectory, but also encode a novel feasible boundary graph to enhance the navigation model's understanding of potential high-value explorable area cues. The navigation decoder includes a feature enhancement module based on a self-attention mechanism, as well as a spatial boundary encoder and a temporal pose encoder based on a cross-attention mechanism. These two sub-decoder modules utilize the three encoded features output from the navigation encoder to construct a long-range spatiotemporal context state representation between the spatial layout and the temporal pose. This invention has achieved high success rates on various indoor scene datasets.

Description

Technical Field

[0001] This invention belongs to the field of embodied intelligence, specifically, it is an indoor robot object navigation method based on a spatiotemporal context-aware converter. Background Technology

[0002] Object navigation, as a key branch of embodied artificial intelligence, provides an important foundation for various visual navigation-related applications, such as image-based target navigation and visual-language navigation. In object navigation, a robot learns to identify the unknown 3D environment from first-person RGB-D images and enables its built-in navigation model to make informed decisions to locate a designated target. Therefore, fully leveraging the spatiotemporal navigation cues obtained from the explored spatial layout and temporal proxy posture trajectories, and using them to accurately guide navigation decisions, has become crucial for the success of object navigation tasks.

[0003] Existing object navigation methods can be broadly categorized into two types: end-to-end methods and modular methods. End-to-end methods directly process first-person perspective RGB images, learning the mapping between the current state and optimal short-term actions to find target objects. While effective, this approach inevitably leads to suboptimal decisions when there are significant differences between the unknown scene layout and prior knowledge, resulting in frequent collisions with surrounding objects or obstacles. In contrast, modular methods utilize scene information accumulated in a bird's-eye view map to determine long-term goals and search for target objects. For example, simple networks like convolutional neural networks (CNNs) are used to fuse accumulated navigation cues, including the explored spatial layout and temporal pose trajectories, to form the robot's current state representation. However, due to the large spatial layouts in different scenes and the inherent local biases of CNNs, simple navigation information fusion based on CNNs may cause the navigation model to ignore the long-term spatiotemporal dependencies between spatial layout and temporal pose trajectories, leading to misunderstandings of the state representation and potential decision failures. Summary of the Invention

[0004] The purpose of this invention is to propose an indoor robot object navigation method based on a spatiotemporal context-aware converter, so as to fully aggregate discriminative spatiotemporal context information to achieve reliable object navigation.

[0005] The technical solution to achieve the purpose of this invention is: an indoor robot object navigation method based on a spatiotemporal context-aware converter, comprising the following steps:

[0006] Step 1: The robot acquires a color image and depth map from the scene from the current first-person perspective. The semantic segmentation network segments different object instances from the color image, and all pixels in the depth map are converted into a 3D point cloud.

[0007] Step two: Align each object pixel category obtained from the semantic segmentation network in step one with the corresponding points in the point cloud, and use geometric transformation to project the point cloud with semantic labels onto the bird's-eye view to obtain the spatial layout and the robot's temporal trajectory.

[0008] Step 3: Use the image processing tool OpenCV to extract the outline of the occupancy map in the spatial layout, and dilate the outer boundary of the obstacle map in the spatial layout. Subtract the processed obstacle map from the occupancy map, and then use a scoring function to filter out important feasible boundaries to construct a feasible boundary map.

[0009] Step 4: Input the feasible boundary map obtained in Step 3, as well as the spatial layout and the robot's temporal trajectory obtained in Step 2, into the navigation encoder to encode the boundary features, spatial layout features, and temporal trajectory features, respectively.

[0010] Step 5: Input the three features obtained in Step 4 into the navigation decoder to construct a more accurate long-range spatiotemporal context state representation, and use this state representation to infer a reasonable long-term goal to guide the robot to explore the unknown scene until the target object is found.

[0011] Furthermore, the semantic segmentation network used in step one is a general residual encoder-decoder network.

[0012] Furthermore, the subtraction of the obstacle map and the occupancy map in step three yields the candidate feasible boundary map. H and W represent the boundary graph M. b Width and height, M b It includes all candidate boundaries, using the scoring function S FO (f i ) for M b The process involves initial screening of all boundaries; the scoring function S FO (f i The working process is as follows:

[0013] S FO (f i ) = A fi +λ·dist(p t ,c fi )#(1)

[0014]

[0015] Where λ represents a constant, p t f represents the robot's current position coordinates. i M represents b One of the boundaries in the middle, Describe the boundary fi The center position coordinates, Describe the boundary f i Number of pixels occupied; merge boundaries with scores higher than a set threshold Th into a single feasible boundary map.

[0016] Furthermore, the spatial layout in step four includes an obstacle map. Occupation Map and an object semantic graph containing k types of objects The robot's temporal trajectory includes the robot's motion trajectory diagram. And the robot's current position map The feasible boundary map M is encoded using the boundary encoder E1 in the navigation encoder. f And obtain boundary features B s Indicates feature b t The number of channels, H2 and W2 represent feature b t Width and height; the spatial layout encoder E2 encodes the obstacle map M. o Occupying map M s and object semantic graph M c The spatial layout, and the characteristics of the spatial layout. K s Representing feature m t Number of channels; E3 timing pose encoder encoding robot motion trajectory diagram M a And the robot's current position map M l and target semantic features Merging yields time-series trajectory features A s Representing feature m t The number of channels.

[0017] Furthermore, in step five, the navigation decoder is used to analyze the boundary feature b. t Spatial layout characteristics m t and time-series trajectory features e t Decoding is performed; firstly, a self-attention mechanism is used to analyze the spatial layout features m. t and time-series trajectory features e t Perform feature enhancement processing:

[0018]

[0019] in These are all learnable parameters. and They are features m t and feature e t After enhancement, the features are sized similarly to m.t and e t The same. In the navigation decoder, the spatial boundary decoder utilizes b t and Calculate the important boundary weights hd3 represents the number of heads in the self-attention mechanism in formula (5):

[0020]

[0021] Timing pose decoder utilizes and Calculate the target-related weights hd4 represents the number of heads in the self-attention mechanism in formula (6):

[0022]

[0023] Important boundary weights W f Target-related weights W s and temporal trajectory features Together they are used to construct the temporal context state representation:

[0024]

[0025] s t After processing through a multilayer perceptron network (MLP), an action distribution for the long-term goal is obtained. Finally, based on a t The probability distribution is used to randomly sample one vertex from the eight vertices of the scene bird's-eye view (including the four corner points and the midpoints of the four boundaries of the bird's-eye view) as the long-term target that the robot needs to explore in the next step. During the process of the robot navigating to the long-term target, if the semantic segmentation network in step two detects the target object, the robot will take the target object as the target point for the next navigation, navigate to the vicinity of the target object, and end the navigation process.

[0026] Compared with the prior art, the present invention has the following significant advantages:

[0027] (1) This invention constructs a novel navigation transformer to effectively aggregate spatiotemporal context navigation cues, providing support for discriminative robot state representation and reliable decision-making.

[0028] (2) This invention develops a navigation encoder enhanced with a drivable boundary map, which is used to encode spatial layout, temporal pose, and spatiotemporal features of the drivable boundary map. In particular, this invention innovatively constructs and integrates an efficient drivable boundary map to identify the most valuable regions in the explored scene.

[0029] (3) The present invention designs a navigation decoder that uses spatial layout, temporal attitude trajectory and traversable boundary to establish a better spatiotemporal context state representation. Attached Figure Description

[0030] Figure 1 This is an overall framework diagram of the method of the present invention.

[0031] Figure 2 It is a schematic diagram for constructing a passable boundary graph.

[0032] Figure 3 This is a comparison diagram between the method of this invention and the object navigation method based on a simple convolutional neural network. Detailed Implementation

[0033] This invention provides an indoor robot object navigation method based on a spatiotemporal context-aware converter, the specific steps of which are as follows:

[0034] Step 1: The robot acquires an RGB color image and a depth map from the scene from the current first-person perspective. The residual encoder-decoder network segments different object instances from the color image and transforms all pixels in the depth map into a 3D point cloud;

[0035] Step 2: Align the pixel categories of each object obtained by the semantic segmentation network in Step 1 with the corresponding points in the point cloud, and use geometric transformation to project the point cloud with semantic labels onto the bird's-eye view to obtain the explored spatial layout and the robot's temporal trajectory.

[0036] Step 3: Use the image processing tool OpenCV to extract the contour of the occupancy map in the spatial layout, and dilate the outer boundary of the obstacle map in the spatial layout. Subtract the processed obstacle map from the occupancy map to obtain the candidate feasible boundary map. H and W represent the boundary graph M. b Width and height. M b It contains all candidate boundaries, and then uses the scoring function S FO (f i ) for M b The process involves initial screening of all boundaries. The scoring function S... FO (f i The working process is as follows:

[0037]

[0038]

[0039] Where λ represents a constant. f represents the robot's current position coordinates. i M represents bOne of the boundaries in the middle, Describe the boundary f i The center position coordinates, Describe the boundary f i Number of pixels occupied. Merge high-scoring boundaries into a single feasible boundary map.

[0040] Step 4: The spatial layout obtained in Step 2 includes an obstacle map. Occupation Map and an object semantic graph containing k types of objects The robot's temporal trajectory includes the robot's motion trajectory diagram. And the robot's current position map The feasible boundary map M is encoded using the boundary encoder E1 in the navigation encoder. f And obtain boundary features B s This represents feature b. t Number of channels; Spatial layout encoder E2 encoding includes obstacle map M o Occupying map M s and object semantic graph M c The spatial layout, and the characteristics of the spatial layout. K s This represents feature m t Number of channels; E3 timing pose encoder encoding robot motion trajectory diagram M a And the robot's current position map M l and target semantic features Merging yields time-series trajectory features A s This represents feature e. t The number of channels.

[0041] Step 5: Use the navigation decoder to analyze boundary feature b t Spatial layout characteristics m t and time-series trajectory features e t Decoding is performed. First, a self-attention mechanism is used to decode the spatial layout features m. t and time-series trajectory features e t Perform feature enhancement processing:

[0042]

[0043] in These are all learnable parameters. In the navigation decoder, the spatial boundary decoder utilizes b... t and Calculate the important boundary weights

[0044]

[0045] Timing pose decoder utilizes and Calculate the target-related weights

[0046]

[0047] Important boundary weights W f Target-related weights W s and time-series trajectory features Together they are used to construct the temporal context state representation:

[0048]

[0049] s t After processing through a multilayer perceptron network (MLP), an action distribution for the long-term goal is obtained. Finally, based on a t The probability distribution is used to randomly sample one vertex from the eight vertices of the scene bird's-eye view (including the four corner points and the midpoints of the four boundaries of the bird's-eye view) as the long-term target that the robot needs to explore in the next step. During the process of the robot navigating to the long-term target, if the semantic segmentation network in step two detects the target object, the robot will take the target object as the target point for the next navigation, navigate to the vicinity of the target object, and end the navigation process.

[0050] The navigation converter in this invention mainly consists of a navigation encoder based on a drivable boundary map and a spatiotemporal navigation decoder. In the navigation encoder, in addition to encoding the spatial layout and temporal position trajectory, this invention also generates an effective drivable boundary map to identify areas where exploration is most likely to find the target. Therefore, this invention encodes this boundary map to enhance the original feature representation with rich exploration guidance. With the above-mentioned encoded spatial and temporal features, the goal of the spatiotemporal navigation decoder in this invention is to learn the spatiotemporal contextual associations between the spatial layout, temporal attitude trajectory, and drivable frontier to achieve reliable navigation information fusion and discriminative state representation. Specifically, a self-attention module is used to enhance the spatial and temporal attributes of the spatial layout and temporal attitude trajectory. Furthermore, based on a cross-attention mechanism, the spatial boundary decoder utilizes the current spatial layout features and boundary features to obtain important boundary weights. The temporal pose decoder utilizes the temporal attitude trajectory features and the current spatial layout features to derive target-related weights. Finally, the spatiotemporal contextual state representation combining important boundary weights and target-related weights is considered to guide the navigation model in making reasonable decisions.

[0051] The present invention will be further described below with reference to the accompanying drawings and embodiments. The described embodiments are merely some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0052] The present invention will now be described in detail with reference to the accompanying drawings. The overall framework of the method is as follows: Figure 1 As shown, an indoor robot object navigation method based on a spatiotemporal context-aware converter is described, and its specific execution steps are as follows:

[0053] Step 1: Figure 1 The RGB color image and depth map on the left side represent the scene information currently acquired by the robot. The residual encoder-decoder network segments different object instances from the color image and transforms all pixels in the depth map into a 3D point cloud.

[0054] Step 2: Figure 1 The left side also shows part of the transformation process of the color images and depth maps collected by the robot into the spatial layout and temporal trajectory from the bird's-eye view. The semantic segmentation network aligns each object pixel category with the corresponding point in the point cloud, and the point cloud with semantic labels is projected onto the bird's-eye view using geometric transformation to obtain the explored spatial layout and the robot's temporal trajectory.

[0055] Step 3: Figure 2 The paper demonstrates part of the construction process of the feasible boundary map. It uses the image processing tool OpenCV to extract the contours of the occupancy map in the spatial layout, and then dilates the outer boundary of the obstacle map in the spatial layout. Finally, it subtracts the processed obstacle map from the occupancy map to obtain the candidate feasible boundary map. H and W represent the boundary graph M. b Width and height. M b It contains all candidate boundaries, and then uses the scoring function S FO (f i ) for M b The process involves initial screening of all boundaries. The scoring function S... FO (f i The working process is as follows:

[0056]

[0057] Where λ represents a constant. f represents the robot's current position coordinates. i M represents b One of the boundaries in the middle, Describe the boundary f i The center position coordinates, Describe the boundary f i Number of pixels occupied. Merge high-scoring boundaries into a single feasible boundary map.

[0058] Step 4: Figure 1 The diagram illustrates the structure and operation of the navigation encoder, and the spatial layout includes an obstacle map. Occupation Map and an object semantic graph containing k types of objects The robot's temporal trajectory includes the robot's motion trajectory diagram. And the robot's current position map The feasible boundary map M obtained in step 1 is encoded using the boundary encoder E1 in the navigation encoder. f And obtain boundary features B s Indicates feature b t The number of channels, H2 and W2 represent feature b t Width and height; the spatial layout encoder E2 encodes the obstacle map M. o Occupying map M s and object semantic graph M c The spatial layout, and the characteristics of the spatial layout. K s Representing feature m t Number of channels; E3 timing pose encoder encoding robot motion trajectory diagram M a And the robot's current position map M l and target semantic features Merging yields time-series trajectory features A s Indicates feature e t The number of channels.

[0059] Step 4: Figure 1 The diagram illustrates the structure and operation of the navigation decoder, utilizing the navigation decoder to analyze boundary features b. t Spatial layout characteristics m t and time-series trajectory features e t Decoding is performed. First, a self-attention mechanism is used to decode the spatial layout features m. t and time-series trajectory features e t Perform feature enhancement processing:

[0060]

[0061] in These are all learnable parameters. In the navigation decoder, the spatial boundary decoder utilizes b... t and Calculate the important boundary weights hd3 represents the number of heads of the self-attention modules in formula (5):

[0062]

[0063] Timing pose decoder utilizes and Calculate the target-related weights hd4 represents the number of heads of the self-attention modules in formula (6):

[0064]

[0065] Important boundary weights W f Target-related weights W s and time-series trajectory features Together they are used to construct the temporal context state representation:

[0066]

[0067] s t After processing through a multilayer perceptron network (MLP), an action distribution for the long-term goal is obtained. Finally, based on a t The probability distribution is used to randomly sample one vertex from the eight vertices of the scene bird's-eye view (including the four corner points and the midpoints of the four boundaries of the bird's-eye view) as the long-term target that the robot needs to explore in the next step. During the process of the robot navigating to the long-term target, if the semantic segmentation network in step two detects the target object, the robot will take the target object as the target point for the next navigation, navigate to the vicinity of the target object, and end the navigation process.

[0068] To further verify the feasibility and effectiveness of the method of this invention, the technical effects of this invention are further described in conjunction with experiments. The hardware platform used in the experiments of this invention is: Intel(R) Xeon(R) Gold 6230 CPU@2.10GHz×32, 80G of memory, and two TITAN GPUs with 24G of video memory each. The software platform used in the experiments of this invention is: Ubuntu 18.04.6LTS operating system and Python 3.7.12 and PyTorch 1.8.1 deep learning framework. Three 3D scenes—Gibson, Habitat-Matterport3D, and Matterport3D—are used to analyze the impact of this method on the robot's object navigation task. Here, the performance of this object navigation model is evaluated using three metrics: success rate (SR), success weighted by path length (SPL), and distance to goal (DTG).

[0069] The method of this invention is used to train in the training scene of Habitat-Matterport3D. During training, the initial position of the robot in the scene and the type of target object are randomly given. The method of this invention was tested in the Habitat-Matterport3D test scenario. To test the performance of the method, the proposed indoor robot object navigation method based on a spatiotemporal context-aware converter was compared with several existing advanced methods. The comparison methods include: the ANS method proposed by Chaplot, DS et al. in their paper "learning to explore using active neural SLAM. In Proceedings of the International Conference on Learning Representations."; the Peanut method for predicting and navigating to unknown targets proposed by Zhai, AJ et al. in their paper "Peanut: Predicting and navigating to unseen targets. In Proceedings of the IEEE / CVF International Conference on Computer Vision, 10926–10935."; and the Zhang, J. et al. method for predicting and navigating to unknown targets proposed in their paper "3d-aware object goal navigation via simultaneous exploration and identification. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition". The 3D-aware three-dimensional perception navigation method proposed in Recognition, 6672–6682; the boundary semantic exploration navigation method FSE proposed by Yu, B. et al. in their paper "Frontier semantic exploration for visual target navigation. In Proceedings of the IEEE International Conference on Robotics and Automation, 4099–4105."; and Cai, W.The pixel-guided navigation method PixNav, proposed by Zhang et al. in their paper "Bridging zero-shot objectnavigation and foundation models through pixel-guided navigation skill. IEEE International Conference on Robotics and Automation, 5228-5234," and the SGM proposed by Zhang et al. in their paper "Imagine Before Go: Self-Supervised Generative Map for Object Goal Navigation. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 16414–16425," are also relevant. Quantitative results are listed in Table 1. Using the intersection-union ratio (IUU) as the evaluation criterion, higher values ​​for the evaluation metrics SR and SPL are better, while lower values ​​for DTG are better.

[0070] Table 1. Quantitative test results on Habitat-Matterport 3D using the method of the present invention and existing technologies.

[0071] method SR (%) SPL (%) DTG (meters) ANS 27.3 9.2 5.80 Peanut 58.8 29.3 3.15 3D-Aware 52.4 24.5 4.25 FSE 53.8 24.6 3.75 PixNav 37.9 20.5 - SGM 59.8 29.4 3.47 Method of the present invention 61.3 29.6 3.40

[0072] As can be seen from the results in Table 1, the method of this invention achieved the best results in all three indicators, indicating that this invention enables robots to better complete object navigation tasks in unfamiliar scenes. Qualitative test results comparing it with other methods are shown below. Figure 3 As shown, compared with the ANS method, the method of the present invention can guide the robot to find the target object by constructing a more ideal long-distance spatiotemporal dependency.

[0073] The above embodiments should be considered as exemplary and non-limiting, but the scope of protection of the present invention is not limited thereto. Any modifications, equivalent substitutions and improvements made by those skilled in the art within the scope of the technology disclosed in the invention and within the spirit and principles of the present invention should be covered within the scope of protection of the present invention.

Claims

1. An indoor robot object navigation method based on a spatiotemporal context-aware converter, characterized in that, Includes the following steps: Step 1: The robot acquires a color image and depth map from the scene from the current first-person perspective, and uses a semantic segmentation network to segment different object instances from the color image. All pixels in the depth map are converted into a 3D point cloud. Step 2: Align the pixel category of each object obtained by the semantic segmentation network in Step 1 with the corresponding point in the point cloud, and use geometric transformation to project the point cloud with semantic labels onto the bird's-eye view to obtain the spatial layout and the robot's temporal trajectory. Step 3: Use the image processing tool OpenCV to extract the outline of the occupancy map in the spatial layout, dilate the outer boundary of the obstacle map in the spatial layout, subtract the processed obstacle map from the occupancy map, and then use a scoring function to filter out important feasible boundaries to construct a feasible boundary map. Step 4: Input the feasible boundary map obtained in Step 3, the spatial layout obtained in Step 2, and the robot's temporal trajectory into the navigation encoder to encode the boundary features, spatial layout features, and temporal trajectory features, respectively. Step 5: Input the three features obtained in Step 4 into the navigation decoder to construct a more accurate long-range spatiotemporal context state representation, and use this state representation to infer a reasonable long-term goal to guide the robot to explore the unknown scene until the target object is found.

2. The indoor robot object navigation method based on a spatiotemporal context-aware converter according to claim 1, characterized in that: The semantic segmentation network used in step one is a general residual encoder-decoder network.

3. The indoor robot object navigation method based on a spatiotemporal context-aware converter according to claim 1, characterized in that: In step three, subtracting the obstacle map from the occupancy map yields the candidate feasible boundary map. H and W represent the boundary graph M. b Width and height, M b It includes all candidate boundaries, using the scoring function S FO (f i ) for M b The process involves initial screening of all boundaries; the scoring function S FO (f i The working process is as follows: Where λ represents a constant, p t f represents the robot's current position coordinates. i M represents b One of the boundaries in the middle, Describe the boundary f i The center position coordinates, Describe the boundary f i Number of pixels occupied; merge boundaries with scores higher than a set threshold Th into a single feasible boundary map.

4. The indoor robot object navigation method based on a spatiotemporal context-aware converter according to claim 1, characterized in that: The spatial layout in step four includes an obstacle map. Occupation Map and an object semantic graph containing k types of objects The robot's temporal trajectory includes the robot's motion trajectory diagram. And the robot's current position map The feasible boundary map M is encoded using the boundary encoder E1 in the navigation encoder. f And obtain boundary features B s Indicates feature b t The number of channels, H2 and W2 represent feature b t Width and height; the spatial layout encoder E2 encodes the obstacle map M. o Occupying map M s and object semantic graph M c The spatial layout, and the characteristics of the spatial layout. K s Representing feature m t Number of channels; E3 timing pose encoder encoding robot motion trajectory diagram M a And the robot's current position map M l and target semantic features Merging yields time-series trajectory features A s Representing feature m t The number of channels.

5. The indoor robot object navigation method based on a spatiotemporal context-aware converter according to claim 1, characterized in that: In step five, the navigation decoder is used to analyze the boundary feature b. t Spatial layout characteristics m t and time-series trajectory features e t Decoding is performed; firstly, a self-attention mechanism is used to analyze the spatial layout features m. t and time-series trajectory features e t Perform feature enhancement processing: in These are all learnable parameters. and They are features m t and feature e t After enhancement, the features are sized similarly to m. t and e t Similarly, in the navigation decoder, the spatial boundary decoder utilizes b t and Calculate the important boundary weights hd3 represents the number of heads in the self-attention mechanism in formula (5): Timing pose decoder utilizes and Calculate the target-related weights hd4 represents the number of heads in the self-attention mechanism in formula (6): Important boundary weights W f Target-related weights W s and time-series trajectory features Together they are used to construct the temporal context state representation: s t After processing through a multilayer perceptron network (MLP), an action distribution for the long-term goal is obtained. Finally, based on a t The probability distribution is used to randomly sample one vertex from the eight vertices of the scene bird's-eye view as the long-term goal that the robot needs to explore in the next step. During the process of the robot navigating to the long-term goal, if the semantic segmentation network in step two detects the target object, the robot will take the target object as the target point for the next navigation, navigate to the vicinity of the target object, and end the navigation process.

6. The indoor robot object navigation method based on a spatiotemporal context-aware converter according to claim 5, characterized in that: The eight vertices of the bird's-eye view of the scene include the four corner points of the bird's-eye view and the midpoints of the four boundaries of the bird's-eye view.

Images