A robot vision-based language navigation method suitable for real indoor environments

Abstract

The present invention relates to a robot visual language navigation method suitable for real indoor environments, comprising the following steps: obtaining environmental information and panoramic images; constructing a two-dimensional grid map of the real indoor environment and converting the continuous indoor environment into a discrete map; generating path-command pair training samples and extracting image feature information from the panoramic images, and constructing a visual language navigation dataset for the real indoor environment based on the two; constructing a visual language navigation model that receives language commands and current environmental information commands and outputs the next action selection at the current moment; performing transfer training on the visual language navigation model using the visual language navigation dataset as training data; and implementing the deployment of the visual language navigation function on a robot based on the ROS system. Compared with existing technologies, the present invention has the advantages of being applicable to indoor environments, significantly improving the success rate of visual language navigation tasks for robots in indoor environments, accelerating network model training, and having strong applicability.

Description

Technical Field

[0001] The present invention relates to the field of robotics and deep learning technologies, and in particular to a robot vision language navigation method suitable for real indoor environments. Background Art

[0002] The ability of robots to understand users' movement instructions in real indoor environments and complete corresponding instructions through autonomous navigation is not only people's vision for the future of artificial intelligence, but also one of the long-term goals that robots need to research.

[0003] Indoor visual language navigation technology requires that after receiving user language commands, the robot continuously observes the visual features of the current scene to output motion signals and ultimately complete the movement task specified in the user's command. Visual language navigation technology mainly includes three main processes: unimodal speech feature understanding, visual feature understanding, and multimodal feature fusion. Speech feature understanding requires the robot to understand the user's command information. Visual feature understanding requires the robot to obtain current environmental information through RGB images and depth images. The multimodal fusion process requires the robot to be able to output the action command that should be selected at the current moment and update the current state information based on the user's command, combined with the current robot state and environmental information.

[0004] Existing indoor visual language navigation models are mainly used in virtual simulation environments, such as Matterport3D and Habitat simulation environments, and are unable to successfully complete experiments in real indoor environments. In actual real-life scenarios, visual language navigation models face two main problems. First, the environmental characteristics of real indoor environments are significantly different from those of virtual simulation environments, and the semantic information of real-life environments is relatively simple, resulting in the inability to directly apply visual language navigation models in virtual environments to real indoor environments. Second, it is impossible to perform continuous environmental modeling on the collected indoor scenes. In other words, it is difficult to construct a similar simulation environment for training based on the indoor environment around you. If the model is trained directly in the real environment, it is necessary to consider the collision cost between the robot and various facilities in the environment, which greatly increases the cost of the experiment. Summary of the Invention

[0005] The purpose of the present invention is to provide a robot vision language navigation method suitable for real indoor environments, so that the robot can realize the vision language navigation function in real indoor environments and keep the experimental cost low.

[0006] The purpose of the present invention can be achieved by the following technical solutions:

[0007] A robot vision-language navigation method suitable for real indoor environments includes the following steps:

[0008] S1. Obtain environmental information and panoramic images collected by the real environment data acquisition hardware platform;

[0009] S2. Construct a two-dimensional grid map of the real indoor environment based on the collected environmental information, classify the room areas of the indoor environment, and regard the current environment as an indoor environment composed of multiple different rooms. The positions and connection relationships of the navigation points of the grid map are defined in the grid map, and the originally continuous indoor environment is converted into a discrete map used for visual language navigation in the real environment. Each navigation point in the discrete map corresponds to a panoramic image.

[0010] S3. Automatically generate path-command pair training samples based on the discrete graph, and manually annotate each generated path to obtain annotation information corresponding to each generated path; perform image feature extraction on the collected panoramic images to obtain image feature information for each navigation point in the discrete graph; and construct a visual language navigation dataset for real indoor environments based on the path-command pair training samples and the image feature information of the panoramic images corresponding to the navigation points in the paths;

[0011] S4. Constructing a visual language navigation model to receive language instructions and current environment information instructions and output the next action selection at the current moment. The visual language navigation model includes a unimodal language feature extraction module, a unimodal environment feature extraction module, and a multimodal feature fusion and state update module;

[0012] S5. Using the visual language navigation dataset as training data, perform transfer training on the visual language navigation model.

[0013] S6. Based on the ROS system, the visual language navigation function is deployed on the robot. The visual language navigation function contains five working nodes, namely the camera node, the voice node, the SLAM node, the robot master node and the network computing node. Among them, the visual language navigation model is deployed on the network computing node.

[0014] The AGV chassis of the real-world data acquisition hardware platform has mapping, autonomous navigation, and positioning capabilities, and the panoramic images captured by the panoramic camera meet viewing angles of 360 degrees horizontally and 90 degrees vertically.

[0015] For a panoramic image, it is segmented at intervals of 30 degrees horizontally and 30 degrees vertically to obtain 36 color RGB images with a resolution of 640*480.

[0016] The path generated in S3 meets three specifications:

[0017] The starting and ending points of the navigation path are located in different rooms in the current environment;

[0018] The navigation path length is greater than the set minimum path length;

[0019] The number of navigation points along the navigation path is greater than the set minimum number of points and less than the set maximum number of points.

[0020] In S3, the collected panoramic pictures are subjected to image feature extraction using the ResNet152 network pre-trained on Image-Net.

[0021] The unimodal language feature extraction module is built based on the BERT-base network and pre-trained using a large corpus dataset; the unimodal environment feature extraction module is built based on the ResNet152 network and pre-trained using ImageNet; the multimodal feature fusion and state update module fuses state, visual, and language information based on a cross-attention mechanism and is pre-trained using visual and language navigation tasks in the simulated environments Matterport3D and habitat.

[0022] The visual language navigation model accepts user instructions at the beginning of the task and uses a unimodal language feature extraction module to obtain language features and initial state features. During the task, the robot receives an image of the current navigation point as input, uses a unimodal environment feature extraction module to obtain the image features of the current navigation point, and obtains the action selection at the current moment and the updated state features through a multimodal feature fusion and state update module.

[0023] During the migration training process of S5, the parameters of the unimodal visual feature extraction model are fixed, no gradient update is performed, and only the network parameters of the unimodal environment feature extraction module and the multimodal feature fusion and state update module are updated.

[0024] The robot master control node is a visual language navigation function process control node, which receives user command information from the voice node, receives coordinate information and issues environment acquisition instructions, obtains the target position of the next navigation point by interacting with the network computing node, and issues navigation instructions to control the movement of the robot; the network computing node receives the environment information and command information issued by the robot master control node, performs network forward calculation, and returns the calculation result, that is, the information of the selected next navigation point.

[0025] Create the message modes VlnNavigation and VlnPoint under ROS for information exchange between the robot master node and the network computing node. Among them, VlnNavigation includes user command information, navigation point information and the number of navigation points, and VlnPoint includes the next navigation point information obtained by the visual language navigation model calculation and the flag whether to stop.

[0026] Compared with the prior art, the present invention has the following beneficial effects:

[0027] (1) The present invention discretizes the indoor environment, which can first simplify the data set acquisition process and accelerate the data construction time. Abstracting the continuous environment into several discrete navigation points and constructing the indoor environment training set through the connection diagram of the navigation points and the panoramic image can omit the three-dimensional reconstruction process of the current environment. At the same time, the present invention automatically generates navigation path samples in the discrete graph in an automated manner, which can quickly generate path samples for training the visual language navigation network model, thereby achieving rapid construction of the training data set.

[0028] (2) For each training sample, the present invention uses manual labeling to label navigation instructions, which has higher accuracy than existing labeling. At the same time, three instructions are labeled for each path, which improves the richness of input instructions.

[0029] (3) The visual language navigation model constructed by the present invention is based on the BERT-base network and the ResNet152 network, and the network model is pre-trained through large-scale corpus datasets, ImageNet datasets, and visual language navigation tasks in the simulation environments Matterport3D and Habitat. Compared with existing networks, the model has higher accuracy and stronger network generalization performance, and can achieve higher accuracy in visual language navigation tasks.

[0030] (4) Based on the visual language navigation model, the present invention uses the data set collected in a discrete indoor environment to use transfer learning to enable the network model to quickly fit the indoor visual language navigation task, thereby accelerating the training and fitting process of the visual language navigation network model and saving a lot of computing power.

[0031] (5) The present invention utilizes the ROS system to deploy visual language navigation functions and adapts the functions of the ROS system, so that the visual language navigation method proposed in the present invention can be widely applied to a large number of mobile robots in the current society, making the deployment of the visual language navigation algorithm more efficient and convenient. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 Schematic diagram of the method flow of the present invention;

[0033] Figure 2 Schematic diagram of a hardware platform for collecting real-world data in an embodiment of the present invention;

[0034] Figure 3 A schematic diagram of a discrete graph used for visual language navigation in an embodiment of the present invention;

[0035] Figure 4A schematic diagram of the actual environment and the generated path in an embodiment of the present invention;

[0036] Figure 5 This is a schematic diagram of the structure of a visual language navigation model in an embodiment of the present invention;

[0037] Figure 6 Schematic diagram of the relationship between working nodes in the ROS system in an embodiment of the present invention. DETAILED DESCRIPTION

[0038] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0039] This embodiment provides a robot vision language navigation method suitable for real indoor environments, such as Figure 1 As shown, the following steps are included:

[0040] S1. Obtain environmental information and panoramic images collected by the real environment data acquisition hardware platform.

[0041] like Figure 2 As shown in the figure, the hardware platform for real-world data acquisition mainly consists of an AGV mobile chassis capable of autonomous positioning and navigation and a panoramic camera, which is used to construct a discrete panoramic map of the real indoor environment. The AGV chassis has the ability to build maps and autonomously navigate and locate, and the panoramic images collected by the panoramic camera meet the viewing angles of 360 degrees horizontally and 90 degrees vertically. For a panoramic image, it is segmented at intervals of 30 degrees horizontally and 30 degrees vertically to obtain 36 color RGB images with a resolution of 640*480. Figure 2 It can be seen that the panoramic camera is fixed on the chassis through a tripod, and the camera is connected to the chassis industrial computer through a USB cable. The panoramic pictures taken will be directly stored in the hard disk of the industrial computer.

[0042] S2. Based on the collected environmental information, a two-dimensional grid map of the real indoor environment is constructed, and the room areas of the indoor environment are classified. The current environment is regarded as an indoor environment composed of multiple different rooms. The navigation point positions and connection relationships of the grid map are defined in the grid map, and the originally continuous indoor environment is converted into a discrete graph G = V, E used for visual language navigation in the real environment. Each navigation point in the discrete graph corresponds to a panoramic image.

[0043] In one embodiment, the grid map and the discrete map are constructed as follows: Figure 3 As shown, the navigation points in the grid map correspond to the coordinate information in the discrete map, and the navigation points are connected by straight lines.

[0044] S3. Automatically generate path-command pair training samples based on the discrete graph G and manually annotate each generated path to obtain the annotation information corresponding to each generated path. Use the ResNet152 network pre-trained on Image-Net to extract image features from the captured panoramic images. Obtain image feature information for each navigation point in the discrete graph to accelerate the training of the visual language navigation model. A visual language navigation dataset for real-world indoor environments is constructed based on the path-command pair training samples and the image feature information of the panoramic images corresponding to the navigation points in the paths.

[0045] In this embodiment, to ensure the quality of the generated path, the generated path must meet three specifications:

[0046] The starting and ending points of the navigation path are located in different rooms in the current environment;

[0047] The navigation path length is greater than the set minimum path length;

[0048] The number of navigation points along the navigation path is greater than the set minimum number of points and less than the set maximum number of points.

[0049] In one embodiment, the actual environment and the generated path are as follows: Figure 4 To increase sample richness and prevent the model from overfitting to the training samples, this example uses three different annotators to annotate the same path, constructing path-command sample pairs for training the visual language navigation model. The annotation process first describes the generated paths in Chinese. Then, the Chinese commands are uniformly translated using a cloud translation interface to generate the corresponding English commands.

[0050] S4. Build a visual language navigation model, receive language instructions and current environment information instructions, and output the next action selection at the current moment.

[0051] In this embodiment, the structure of the visual language navigation model is as follows: Figure 5 As shown, it includes a unimodal language feature extraction module, a unimodal environment feature extraction module and a multimodal feature fusion and state update module.

[0052] The unimodal language feature extraction module is built based on the BERT-base network and pre-trained using a large corpus dataset; the unimodal environment feature extraction module is built based on the ResNet152 network and pre-trained using ImageNet; the multimodal feature fusion and state update module uses a cross-attention mechanism to fuse state, visual and language information, and is pre-trained using visual and language navigation tasks in the simulated environments Matterport3D and habitat.

[0053] The visual language navigation model accepts user instructions at the beginning of the task and uses the unimodal language feature extraction module to obtain language features and initial state features. During the task, the robot receives the image of the current navigation point as input, uses the unimodal environment feature extraction module to obtain the image features of the current navigation point, and obtains the current action selection and updated state features through the multimodal feature fusion and state update module.

[0054] S5. Use the visual language navigation dataset as training data to perform transfer training on the visual language navigation model.

[0055] During the transfer training process, the parameters of the unimodal visual feature extraction model are fixed and no gradient update is performed. Only the network parameters of the unimodal environment feature extraction module and the multimodal feature fusion and state update module are updated.

[0056] At the same time, a smaller learning rate is used during training to prevent the model from overfitting.

[0057] S6. Based on the ROS system, the visual language navigation function is deployed on the robot. The visual language navigation function contains five working nodes, namely the camera node, voice node, SLAM node, robot master node and network computing node.

[0058] like Figure 6 As shown, the robot master node is the visual language navigation function process control node, which receives user command information from the voice node, receives coordinate information and issues environment acquisition instructions, obtains the target position of the next navigation point by interacting with the network computing node, and issues navigation instructions to control the robot movement; the visual language navigation model is deployed on the network computing node, the network computing node receives the environment information and command information issued by the robot master node, performs network forward calculation on the mobile computer, and returns the calculation result, that is, the selected next navigation point information.

[0059] Create the message modes VlnNavigation and VlnPoint under ROS for information exchange between the robot master node and the network computing node. Among them, VlnNavigation includes user command information, navigation point information and the number of navigation points, and VlnPoint includes the next navigation point information obtained by the visual language navigation model calculation and the flag whether to stop.

[0060] The actual use process of the robot vision language navigation function is as follows Figure 1As shown on the right, the service robot first waits for user commands. Upon receiving a valid command, it calls a cloud interface for text translation, converting the received voice information into English text and beginning the current visual language navigation task. After the task begins, the robot captures a panoramic image at the navigation point and extracts image information of candidate points corresponding to the current panoramic image based on the indoor discrete map and the current actual orientation. The robot then sends the user command and the candidate point image information via a message to the mobile computer for network forward calculation. If the network forward calculation result does not result in a stop signal, the sequence number of the corresponding next navigation point is assigned. The specific location of the next navigation point is then determined using the discrete map. The robot then calls the SLAM system interface to move to the next navigation point and waits for the arrival signal from the SLAM system. When the robot reaches the next navigation point, the environment acquisition process repeats until the network forward calculation outputs a stop signal, indicating that the navigation task corresponding to the current command has been completed and the visual language navigation function process ends.

[0061] The above describes in detail the preferred embodiments of the present invention. It should be understood that those skilled in the art can make numerous modifications and variations based on the concepts of the present invention without inventive effort. Therefore, any technical solutions that can be derived by those skilled in the art through logical analysis, reasoning, or limited experimentation based on the concepts of the present invention and the prior art should be within the scope of protection defined by the claims.

Claims

1. A robot vision language navigation method suitable for real indoor environments, characterized by: The following steps are involved: S1. Obtain environmental information and panoramic images collected by the real environment data acquisition hardware platform; S2. Construct a two-dimensional grid map of the real indoor environment based on the collected environmental information, classify the room areas of the indoor environment, and regard the current environment as an indoor environment composed of multiple different rooms. The positions and connection relationships of the navigation points of the grid map are defined in the grid map, and the originally continuous indoor environment is converted into a discrete map used for visual language navigation in the real environment. Each navigation point in the discrete map corresponds to a panoramic image. S3. Automatically generate path-command pair training samples based on the discrete graph, and manually annotate each generated path to obtain annotation information corresponding to each generated path; perform image feature extraction on the collected panoramic images to obtain image feature information for each navigation point in the discrete graph; and construct a visual language navigation dataset for real indoor environments based on the path-command pair training samples and the image feature information of the panoramic images corresponding to the navigation points in the paths; S4. Constructing a visual language navigation model to receive language instructions and current environment information instructions and output the next action selection at the current moment. The visual language navigation model includes a unimodal language feature extraction module, a unimodal environment feature extraction module, and a multimodal feature fusion and state update module; S5. Using the visual language navigation dataset as training data, perform transfer training on the visual language navigation model. S6. Based on the ROS system, the visual language navigation function is deployed on the robot. The visual language navigation function contains five working nodes, namely the camera node, the voice node, the SLAM node, the robot master node and the network computing node. Among them, the visual language navigation model is deployed on the network computing node.

2. The robot vision language navigation method applicable to a real indoor environment according to claim 1, characterized in that: The AGV chassis of the real-world data acquisition hardware platform has mapping, autonomous navigation, and positioning capabilities, and the panoramic images captured by the panoramic camera meet viewing angles of 360 degrees horizontally and 90 degrees vertically.

3. The robot vision language navigation method applicable to a real indoor environment according to claim 2, characterized in that: For a panoramic image, it is segmented at intervals of 30 degrees horizontally and 30 degrees vertically to obtain 36 color RGB images with a resolution of 640*480.

4. The robot vision language navigation method applicable to a real indoor environment according to claim 1, characterized in that: The path generated in S3 meets three specifications: The starting and ending points of the navigation path are located in different rooms in the current environment; The navigation path length is greater than the set minimum path length; The number of navigation points along the navigation path is greater than the set minimum number of points and less than the set maximum number of points.

5. The robot vision language navigation method applicable to a real indoor environment according to claim 1, characterized in that: In S3, the collected panoramic pictures are subjected to image feature extraction using the ResNet152 network pre-trained on Image-Net.

6. The robot vision language navigation method applicable to a real indoor environment according to claim 1, characterized in that: The unimodal language feature extraction module is built based on the BERT-base network and pre-trained using a large corpus dataset; the unimodal environment feature extraction module is built based on the ResNet152 network and pre-trained using ImageNet; the multimodal feature fusion and state update module fuses state, visual, and language information based on a cross-attention mechanism and is pre-trained using visual and language navigation tasks in the simulated environments Matterport3D and habitat.

7. The robot vision language navigation method applicable to a real indoor environment according to claim 1, characterized in that: The visual language navigation model accepts user instructions at the beginning of the task and uses a unimodal language feature extraction module to obtain language features and initial state features. During the task, the robot receives an image of the current navigation point as input, uses a unimodal environment feature extraction module to obtain the image features of the current navigation point, and obtains the action selection at the current moment and the updated state features through a multimodal feature fusion and state update module.

8. The robot vision language navigation method applicable to a real indoor environment according to claim 1, characterized in that: During the migration training process of S5, the parameters of the unimodal visual feature extraction model are fixed, no gradient update is performed, and only the network parameters of the unimodal environment feature extraction module and the multimodal feature fusion and state update module are updated.

9. The robot vision language navigation method applicable to a real indoor environment according to claim 1, characterized in that: The robot master control node is a visual language navigation function process control node, which receives user command information from the voice node, receives coordinate information and issues environment acquisition instructions, obtains the target position of the next navigation point by interacting with the network computing node, and issues navigation instructions to control the movement of the robot; the network computing node receives the environment information and command information issued by the robot master control node, performs network forward calculation, and returns the calculation result, that is, the information of the selected next navigation point.

10. The robot vision language navigation method applicable to a real indoor environment according to claim 1, characterized in that: Create the message modes VlnNavigation and VlnPoint under ROS for information exchange between the robot master node and the network computing node. Among them, VlnNavigation includes user command information, navigation point information and the number of navigation points, and VlnPoint includes the next navigation point information obtained by the visual language navigation model calculation and the flag whether to stop.

Images